Geek as a platform

Spam via Google Analytics

I have to give it to them... People are pretty damn creative... Email spam, Whatsapp spam, and why not... Referral spam!

'spam'

People should put all this effort and time in making the world a better place

Minimal Cygwin Python data science installation

A few days ago a made a fresh installation of Cygwin64 in order to tinker with Pandas (just for fun) but instead of installing Anaconda, I decided to install each component manually... I will write it here because I'm sure I will need it in the future (and it might be handy for someone else).

First things first, package management in Cygwin is a bit of a pain without apt-cyg... So download it from here, save it as "apt-cyg" in your $HOME and install it:

install apt-cyg /bin/

Start installing stuff:

apt-cyg install wget curl gcc-core g++ libzmq-devel libzmq5 python3 python3-zmq libfreetype-devel

Download and install pip:

wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py

Some of the libraries included in cygwin repository are a little bit old, so it is better to download and compile them from pip... ie:

Cython (0.24.1)  - The Cython compiler for writing C extensions for the Python language.
  INSTALLED: 0.22
  LATEST:    0.24.1

Install python stuff from pip:

pip install cython
pip install jupyter
pip install numpy
pip install pandas
pip install matplotlib

And that's it... It just take a few minutes to have everything installed and compiled, but it is not too complicated.

psgonza@host$ ipython
Python 3.4.3 (default, May  5 2015, 17:58:45)
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import pandas as pd

One remark, in this installation, there is no Fortran support, so in case you need it, you'd need to compile ATLAS, LAPACK, etc.. Last time I did it a couple of years ago, it took me a while, and I'd say I have never really used it, so this time I skipped it. If you really need them, Cygwin might not be what you should be using (IMO)...

Later!

PS: Using Anaconda is a perfectly valid option as well, but it is built for Windows, so you need to link it with your cygwin python installation. You can check this link in case you decide to go that way.

Running telegram-cli in cygwin

Telegram is a really interesting messaging application... It is secure (well, let's say secure enough), fast and multiplatform... Even there is a CLI client!

I tried to compile the telegram-cli in my cygwin64 earlier today following their documentation, and I came across a couple of issues.

1) Patching failed:

I couldn't patch Makefile and Hunk #1 failed in loop.c as well:

20:34:26@tg$ patch -p1 < telegram-cli-cygwin.patch
patching file Makefile
Hunk #1 FAILED at 4.
1 out of 1 hunk FAILED -- saving rejects to file Makefile.rej
patching file loop.c
Hunk #1 FAILED at 383.
Hunk #2 succeeded at 737 with fuzz 2 (offset 103 lines).
1 out of 2 hunks FAILED -- saving rejects to file loop.c.rej

Probably the patch file is not updated with the latest telegram-cli changes, so I dediced to patch it manually...

I updated the first lines of Makefile with the changes stated in the patch file:

1 srcdir=.
2
3 CFLAGS=-g -O2
4 LDFLAGS= -L/usr/local/lib -L/usr/lib -L/usr/lib -L/usr/lib
5 CPPFLAGS= -I/usr/local/include -I/usr/include -I/usr/include  -I/usr/include/python3.4m -I/usr/include
6 DEFS=-DHAVE_CONFIG_H
7
8 COMPILE_FLAGS=${CFLAGS} ${CPFLAGS} ${CPPFLAGS} ${DEFS} -Wall -Werror -Wextra -Wno-missing-field-initializers 
   -Wno-deprecated-declarations -fno-strict-aliasing -fn      o-omit-frame-pointer -ggdb -Wno-unused-parameter
9 EXTRA_LIBS=-ljansson -lconfig -lz -levent -lm   -lreadline -llua-5.2  -lpython3.4m -lssl -lcrypto
10 LOCAL_LDFLAGS=-ggdb -levent ${EXTRA_LIBS} -ldl -lpthread -lutil
11 LINK_FLAGS=${LDFLAGS} ${LOCAL_LDFLAGS}

Since loop.c was partially patched already, I tried to manually fix Hunk #1, but nothing worked, so I guess it is no longer neccessary...

I managed to generate the binary file after patching Makefile... But it still refused to work.

2) Error assertion "0" failed: file "tgl/mtproto-utils.c", line 101, function: BN2ull

There were no compilation errors, but there was something wrong in tgl/mtproto-utils.c:

21:00:14@tg$ bin/telegram-cli -k tg-server.pub
Telegram-cli version 1.4.1, Copyright (C) 2013-2015 Vitaly Valtman
Telegram-cli comes with ABSOLUTELY NO WARRANTY; for details type `show_license'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show_license' for details.
Telegram-cli uses libtgl version 2.1.0
Telegram-cli includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit. (http://www.openssl.org/)
I: config dir=[/home/user/.telegram-cli]
[/home/user/.telegram-cli] created
[/home/user/.telegram-cli/downloads] created
> SIGNAL received
No libexec. Backtrace disabled
assertion "0" failed: file "tgl/mtproto-utils.c", line 101, function: BN2ull

The solution seems to be remove (or comment out) lines 101 and 115 in tgl/mtproto-utils.c:

[...]
101     //assert (0); // As long as nobody ever uses this code, assume it is broken.
[...]
115     //assert (0); // As long as nobody ever uses this code, assume it is broken.
[...]

After that, run make again, and execute it:

21:13:09@tg$ bin/telegram-cli -k tg-server.pub
Telegram-cli version 1.4.1, Copyright (C) 2013-2015 Vitaly Valtman
Telegram-cli comes with ABSOLUTELY NO WARRANTY; for details type `show_license'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show_license' for details.
Telegram-cli uses libtgl version 2.1.0
Telegram-cli includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit. (http://www.openssl.org/)
I: config dir=[/home/user/.telegram-cli]
>

I hope it helps!

\\psgonza

Tamron SP AF10-24mm

I am not a pro photographer and don't like to spend a fortune in gear for my camera, but I've always wanted to buy an ultrawide lense...

I had the opportunity to buy the Tamron SP AF10-24mm for a decent price, and so far I am liking it... See a couple of pics I took a few days ago:

'Shibuya crossing'

'Zenko-ji Nagano'

'Matsumoto Castle'

Maybe not the best uw lense out there, but good enough for the price

\\psgonza

We love you Bash

We love you Bash… but sometimes you are so damn slow

I'm sure this has happened to you before… I had to create an ldif file containing 70.000 entries like these ones:

dn: user=XXXXXXXXXX,rootdn=com
changetype: modify
replace: attr1
attr1: qwerty1
-
replace: attr2
attr2: {"reporting": [{"name":"Total","reportingLevel":"total","subscription":"A", "time":500,"reset":{"main":"30 ","time":"30 "}}}]}

(the real data was a little bit larger, this is just an example)

It was supposed to be something really quick and simple, one of those one-liners you write in 3 seconds… I said to myself: bash + loop + echo + redirection to file = WIN

Well, it proved me wrong :

The quick and dirty code to be executed in the shell:

for i in `seq 1 70000`;
do
echo -e "dn: user=XXXXXXXXXX,rootdn=com
changetype: modify
replace: attr1
attr1: qwerty1
-
replace: attr2
attr2: {\"reporting\": [{\"name\":\"Total\",\"reportingLevel\":\"total\",\"subscription\":\"A\", \"time\":500,\"reset\":{\"main\":\"30 \",\"time\":\"30 \"}}}]}
"  >> modify.ldif
done

So I issued that command, and went out for a bite… 30 minutes later, the fraking thing was still running!!! How was that even possible…

I realized I was opening the file every time I wrote a line, a nonsense… so I just stopped the script and tweaked it a little bit… Result: pretty much the same… Here's some figures with only 5000 iterations per loop with different redirections:

Initial redirection:

for i in `seq 1 5000`; do

> "  >> modify.ldif; done
real    4m17.606s
user    1m4.028s
sys     1m35.850s

Only one redirection outside of the loop:

for i in `seq 1 5000`; do

> " ; done >> modify.ldif
real    4m30.218s
user    1m8.496s
sys     1m41.390s

I also tried to use a file descriptor, just for the sake of it, but it has no impact:

exec 4>> modify.ldif
for i in `seq 1 5000`; do

> "  >&4; done
real    4m32.197s
user    1m10.248s
sys     1m39.294s

And using the FD outside of the loop:

for i in `seq 1 5000`; do

> "; done >&4
real    4m31.478s
user    1m11.740s
sys     1m38.678s

So all in all, as you can see, Bash IO redirection performance, basically sucks.

A 20 lines python script (with no optimization whatsoever) is able to generate the whole ldif file (creating 70.000 entries to modify) in less than 1 second:

#!/usr/bin/env python

data="""dn: user=XXXXXXXXXX,rootdn=com
changetype: modify
replace: attr1
attr1: qwerty1
-
replace: attr2
attr2: {\"reporting\": [{\"name\":\"Total\",\"reportingLevel\":\"total\",\"subscription\":\"A\", \"time\":500,\"reset\":{\"main\":\"30 \",\"time\":\"30 \"}}}]}
"""

with open("modify.ldif",'w') as outputF:
    for iter in xrange(70000):
        outputF.write(data.replace("XXXXXXXXXX ",str(iter)))

Executing it:

# time python gen_ldif.py
real    0m0.506s
user    0m0.108s
sys     0m0.052s

How about that? Sometimes it's just better to use the right tool for the right job...

So yeah, we love you Bash… Even though you are really slow sometimes.

\\psgonza