My talk @ EuroPython 2016

I had the great ex­pe­rien­ce of pre­sen­ting at Eu­ro­P­y­thon 2016. My ta­lk en­ti­tled “Clean co­de in Py­tho­n”, was about good de­ve­lo­p­ment prac­ti­ce­s, do­wn to lo­w-­le­vel de­sign (wi­th co­de exam­ple­s), for Py­tho­n. The idea of the ta­lk, was to pre­sent the “p­y­tho­ni­c” appro­ach for wri­ting co­de, and how do ge­ne­ral con­cep­ts of ­clean co­de apply to Py­tho­n.

The­se exam­ples mi­ght be use­ful for be­gin­ner­s, de­ve­lo­pers ex­pe­rien­ced in other lan­gua­ges co­ming to Py­tho­n, and peo­ple using Py­thon for scien­ti­fic appli­ca­tion­s. The exam­ples could al­so be hel­pful for se­nio­r ­de­ve­lo­per­s, be­cau­se they re­mind real si­tua­tions that mi­ght appear in pu­ll re­ques­ts, whi­le doing a co­de re­view.

He­re is the vi­deo on You­Tu­be:

And the sli­des (whi­ch I al­so ma­de avai­la­ble along wi­th the sour­ce co­de, shor­tly after the pre­sen­ta­tio­n ­fi­nis­hed).

The pre­sen­ta­tion was we­ll re­cei­ve­d: so­me atten­dees ­told me they liked it (e­ven asked for the sli­des and co­de), and I got good ad­vi­ce­s. ­The fo­llo­wing da­ys of the con­fe­ren­ce, mo­re peo­ple told me that they liked the pre­sen­ta­tio­n, and so­me other­s ­men­tio­ned (so­me­thing I did not thi­nk at the be­gin­nin­g, but that it makes per­fect sen­se), that the­se idea­s a­re rea­lly use­ful for peo­ple using Py­thon in scien­ti­fic en­vi­ron­men­ts.

I am glad it was use­ful for the co­m­mu­ni­ty.

EuroPython 2016 remarks

Last week, Eu­ro­P­y­thon 2016 fi­nis­he­d, and it was an ama­zing con­fe­ren­ce I had the plea­su­re to atten­d. He­re is my re­view of tho­se da­ys.

The conference

I arri­ved on Satur­day noon at Bil­bao, Spai­n, the day be­fo­re the con­fe­ren­ce, so I had so­me ti­me to kno­w ­the ci­ty, see the ve­nues, etc. The next da­y, on Sun­da­y, was for two se­pa­ra­te wo­rks­hop­s: Djan­go girls an­d ­Be­gin­ne­r’s da­y. I atten­ded the be­gin­ne­r’s day as a co­ach, and hel­ped a group of in­ter­me­dia­te de­ve­lo­pers wi­th se­ve­ral exer­ci­ses ai­med at ex­plai­ning so­me Py­thon con­cep­ts, su­ch as: con­text ma­na­ger­s, de­co­ra­tor­s, ­ma­gic me­tho­d­s, ge­ne­ra­tor­s, etc. It was rea­lly cu­rious that so­me of the­se to­pi­cs we­re tho­se I was going to co­ve­r on my ta­lk on Wed­nes­da­y, so I felt rea­lly glad about tha­t. I took an oa­th (ve­ry funny BTW) for be­co­ming a be­gin­ne­r’s men­to­r, and so I di­d (it was rea­lly good ac­tua­ll­y). I had a great ti­me hel­ping other de­ve­lo­per­s, ex­chan­ging ideas and ex­pe­rien­ces du­ring lun­ch, ­sol­ving pro­ble­ms, and ge­tting a first glimp­se on what the con­fe­ren­ce was going to be like.

../ep2016-beginners-mentor-oath.png

The mo­ment of the oa­th for be­co­ming a men­to­r, and ear­ning the ba­dge.

After the wo­rks­hop fi­nis­he­d, I wa­lked to the main ve­nue, and ga­ve a hand pa­cking the bags of the con­fe­ren­ce. After tha­t, ti­me to­ ­see around Bil­bao.

From Mon­day to Fri­day was the con­fe­ren­ce itsel­f, wi­th all the ta­lks, and tra­i­nings.

Mon­day started wi­th the in­tro­duc­tion to the con­fe­ren­ce, and shor­tly the­rea­fte­r, the ve­ry first ke­y­no­te by Ra­chel Wi­ll­me­r, who ga­ve a great pre­sen­ta­tio­n, sha­ring a lot of ex­pe­rien­ce, and in­te­res­ting idea­s.

At around noon the­re was a ke­y­no­te by N. To­ller­vey about Mi­cro­P­y­tho­n. The pre­sen­ta­tion was ex­ce­llent (o­ne of the ones I liked the mos­t), and the idea of the pro­ject is awe­so­me. On top of tha­t, it was an­noun­ced that the BBC wa­s ­gi­ving away mi­cro­:­bi­ts for the atten­dees of ­the con­fe­ren­ce, so it was a great sur­pri­se to pi­ck up mi­ne at the con­fe­ren­ce desk. I even started pla­ying around a bit wi­th it (mo­re in a fu­tu­re pos­t).

The rest of the after­noon, I atten­ded se­ve­ral ta­lks. At the en­d, the­re we­re, of cour­se the li­gh­tning ta­lks, whi­ch we­re ama­zin­g.

Tues­day started wi­th the ke­y­no­te by P. Hil­de­bran­t, pre­sen­ting how Dis­ney uses se­ve­ral te­ch­no­lo­gie­s, in­clu­ding Py­tho­n, as su­pport for mo­vies and pro­duc­tion­s. It was ve­ry good and en­li­gh­ten­ing to see an en­dea­vour of su­ch ex­tent wi­th Py­tho­n. A­fter tha­t, du­ring mor­ning I atten­ded a wo­rks­hop about As­ync web de­ve­lo­p­men­t, wi­th se­ve­ral Py­thon te­ch­no­lo­gie­s ­for doing as­yn­ch­ro­nous com­pu­ta­tion.

Du­ring the after­noon, I wa­tched se­ve­ral great ta­lks, in­clu­ding “Pro­tect you users wi­th Cir­cuit Breaker­s”, an­d se­ve­ral other good ones, clo­sing wi­th the li­gh­tning ta­lks.

Wed­nes­day was the day of my ta­lk, so I atten­ded so­me ta­lks du­ring mor­ning and then, at the after­noon, I pre­sen­ted mi­ne. I rea­lly liked how it de­ve­lo­pe­d. Mo­reo­ve­r, it was rea­lly good to re­cei­ve good fee­dba­ck from so­me atten­dees, sa­ying the­y ­liked it, and that it was use­ful for the­m. Shor­tly the­rea­fte­r, I pu­blis­hed the sli­des and the sour­ce co­de.

On Thurs­da­y, the­re we­re so­me ta­lks about as­yn­c/await and as­yn­ch­ro­nous pro­gra­m­ming in Py­thon 3, mo­cks, and hi­gh-a­vai­la­bi­li­ty ar­chi­tec­tu­re.

On Fri­da­y, the ke­y­no­te was about how Py­thon is us­ed by the scien­ti­fic co­m­mu­ni­ty. It was ve­ry en­li­gh­ten­in­g, and in­te­res­tin­g ­to see ano­ther use ca­se of Py­tho­n, and how is be­co­ming the main te­ch­no­lo­gy on this area.

The ta­lks du­ring mor­ning in this ca­se, we­re di­vi­ded among se­ve­ral to­pi­cs, being the main ones: ins­tru­men­ta­tion for per­for­man­ce ­me­tri­cs, “How to mi­gra­te form Pos­tgreS­QL to HDF5 and li­ve ha­ppi­ly ever afte­r”, “S­plit Up! Fi­gh­ting the mo­n­oli­th”. Du­ring the after­noon, I joi­ned a wo­rks­hop about Do­cke­r, on whi­ch we built an appli­ca­tion using Do­cke­r-­com­bi­ne, and fo­llo­wed good prac­ti­ce­s.

It is wor­th men­tio­nin­g, that on Fri­day the­re was an spe­cial edi­tion for li­gh­tning ta­lks, whi­ch was not in the ori­gi­nal sche­du­le. Afte­r ­making so­me arran­ge­men­ts, and due to so­me on-­the-­fly chan­ges, it was po­s­si­ble to ha­ve ano­ther ses­sion for li­gh­tning ta­lks, ri­ght be­fo­re ­the sprin­ts orien­ta­tion and the clo­sing ses­sio­n.

Satur­day and Sun­day we­re for sprin­ts (ha­cka­thon­s). On Satur­day I joi­ned to sprint on aioh­ttp, and ac­tua­ll­y ­su­bmi­tted a pu­ll re­quest, tha­t was mer­ge­d, whe­reas on Sun­day I wanted to che­ck on a py­test is­sue.

My talk

It was great to ha­ve the oppor­tu­ni­ty to pre­sent at Eu­ro­P­y­tho­n. What was even be­tte­r, was the po­si­ti­ve fee­dba­ck I got from other atten­dees, and the fact that it was use­ful and in­te­res­ting for them (whi­ch wa­s, in the en­d, what I ca­red most abou­t). I found the ex­pe­rien­ce ve­r­y ­po­si­ti­ve.

From the co­m­men­ts, I ga­the­red so­me­thing I ha­ve not no­ti­ced when I first en­vi­sio­ned the ta­lk, whi­ch is how use­ful the­se con­cep­ts mi­gh­t ­be for peo­ple using Py­thon for scien­ti­fic appli­ca­tion­s. It see­ms, scien­tis­ts using Py­thon for da­ta pro­ce­s­sing or com­pu­ta­tio­n, do no­t u­sua­lly ha­ve the ba­ck­ground of a de­ve­lo­pe­r, so con­cep­ts like co­de rea­da­bi­li­ty, te­ch­ni­cal deb­t, and main­tai­na­bi­li­ty, are hel­pful in or­de­r ­to im­pro­ve the co­de ba­se. This ga­ve me the idea of adap­ting the exam­ple­s, perhaps adding one re­lated to the­se area­s.

Python use cases

The­re we­re peo­ple from many coun­trie­s, in­dus­trie­s, and com­pa­nies wi­th di­ffe­rent ba­ck­groun­d­s. The trend see­ms to be now on ­da­ta scien­ce, but Py­thon is wi­de­ly us­ed in many area­s.

I be­lie­ve the main areas of fo­cus for Py­thon are: so­ftwa­re de­ve­lo­p­men­t, sys­tem ad­mi­nis­tra­tion / Dev Op­s, and scien­ce.

The­re we­re ta­lks, tra­cks, ses­sion­s, and tra­i­nings for all of the­m, wi­th ve­ry te­ch­ni­cal de­tai­l.

Highlights

The­re we­re so many great ta­lks and re­sour­ces that I can­not na­me ea­ch sin­gle one of the­m, so I wi­ll point the mai­n ­to­pi­cs and so­me of the ta­lks that gra­bbed my atten­tion the mos­t, but plea­se keep in mind that all we­re grea­t.

Among the many things pen­ding to test and re­sear­ch, are al­so books. I lear­ned about PY­RO­4, for ma­na­ging Py­thon re­mo­te ob­jec­ts, whi­ch see­ms like a pro­mi­sing te­ch­no­lo­g­y. I wi­ll di­ve in­to mo­re de­tail on con­da and the buil­ding sys­te­ms, con­da ­chan­nel­s, etc. The ta­lk “Ex­plo­ring your Py­thon in­ter­pre­te­r” was rea­lly in­te­res­tin­g, and it was a good in­tro­duc­tio­n, in or­de­r ­to be­co­me in­vol­ved wi­th CP­y­thon de­ve­lo­p­men­t.

I atten­ded many ta­lks about the la­test fea­tu­res of Py­thon 3.5, su­ch as as­yn­cIO, co­rou­ti­nes, and all the new func­tio­na­li­ties fo­r a­s­yn­ch­ro­nous pro­gra­m­min­g, and they all we­re rea­lly in­te­res­tin­g. In par­ti­cu­lar “The re­port of Twis­te­d’s Dea­th” was ve­ry in­te­res­tin­g, an­d (s­poi­ler aler­t), it looks like sti­ll has an in­te­res­ting fu­tu­re com­pe­ting wi­th the new li­bra­ries and stan­dar­d­s.

On the li­gh­tning ta­lks, it was pre­sen­ted a re­ver­se de­bu­gger (re­v­db), and its de­mo was ama­zin­g.

Conclusion

After atten­ding many ta­lks, and tra­i­nings, ta­lking to many other ex­pe­rien­ce de­ve­lo­per­s, sys­tem ad­mi­nis­tra­tor­s, and da­ta scien­tis­ts, I can sta­te that the con­fe­ren­ce has an ama­zing lear­ning en­vi­ron­men­t, and the ou­tco­me was com­ple­te­ly po­si­ti­ve. It was use­fu­l ­for ca­tching up wi­th te­ch­no­lo­g­y, che­cking the en­vi­ron­ment and see how Py­thon is being us­ed or de­plo­yed in the wil­d, learn fro­m u­se ca­ses, ex­pe­rien­ce­s, and ex­chan­ge idea­s.

The con­tent was rea­lly ins­pi­ring and open-­min­din­g. I ha­ve lo­ts of ite­ms to che­ck, as poin­ts for re­sear­ch, whi­ch I wi­ll co­ver in fo­llo­win­g en­trie­s.

Py­thon 3 is mu­ch mo­re wi­de­ly us­ed than one would ex­pec­t. It is ac­tua­lly the stan­dard no­w, and many ta­lks (in­clu­ding mi­ne), we­re using Py­thon 3 ­co­de, but most im­por­tan­tl­y, most pro­jec­ts are now in this ver­sio­n, whe­reas Py­thon 2 looks like the le­ga­cy op­tio­n. Good news :-)

All in all, this edi­tion of Eu­ro­P­y­thon was awe­so­me, and I am looking fo­rward to pre­sen­ting again next yea­r!

Upcoming talk at EuroPython 2016

I am glad to in­form that I wi­ll be speaking at Eu­ro­P­y­thon 2016 con­fe­ren­ce.

My su­bmis­sion about clean co­de in Py­thon was ac­cep­te­d, so in the next edi­tion of Eu­ro­P­y­thon 2016, in Bil­bao, Spai­n, I wi­ll ta­lk about clean co­de prin­ci­ples for Py­tho­n. He­re is the abs­trac­t:

https://ep2016.europython.eu/conference/talks/clean-code-in-python

The fu­ll list of ta­lks is avai­la­ble at:

https://ep2016.europython.eu/en/events/sessions/

If you are in­te­res­te­d, subs­cri­be to the Eu­ro­P­y­thon blog and You­tu­be chan­ne­l. I wi­ll in­clu­de mo­re de­tail­s in a se­pa­ra­te pos­t.

Glimpses of a Vim configuration

It’s been a whi­le sin­ce I started tra­cking ver­sions of my cus­tom Vim con­fi­gu­ra­tio­n, and making it avai­la­ble as an open sour­ce so­ftwa­re in Gi­thub. ­The best of this pro­ject is, in my opi­nio­n, to ha­ve it un­der ver­sion con­tro­l, so I can tra­ck chan­ges and re­lea­ses.

Eve­ry on­ce in a whi­le, when I find a new se­ttin­g, or a great new fea­tu­re, I mo­di­fy the con­fi­gu­ra­tio­n, so they wi­ll be­co­me avai­la­ble on the next re­lea­se. ­Be­si­des the fea­tu­res that are men­tio­ned in the pro­jec­t, and the cus­to­mi­za­tions ma­de, I feel ve­ry co­m­for­ta­ble wi­th the co­lour sche­me I ma­de.

He­re are so­me glimp­ses of it:

../vim-capture1.png

First cap­tu­re of co­lour­s, and la­yout

The co­lour sche­me is ge­ne­ral for the syn­tax hi­gh­li­gh­ting of all ty­pes re­cog­ni­zed by Vi­m. Plea­se no­te this mi­ght al­so de­pend on the con­fi­gu­ra­tion of your ter­mi­na­l.

../vim-capture2.png

The ta­bs are al­so the­med ac­cor­ding to the me­nus.

Any su­gges­tions or im­pro­ve­men­ts to the co­de and con­fi­gu­ra­tion can be ma­de on the Gi­thub pro­jec­t.

Deleting commented out code

This is a ru­le I alwa­ys en­cou­ra­ge in so­ftwa­re de­ve­lo­p­men­t. Mo­reo­ve­r, I con­si­der it to be so­me­thing that ha­s ­to be in­clu­ded in eve­ry co­ding gui­de­li­ne of a good pro­jec­t.

The­re are se­ve­ral rea­sons for this, but pro­ba­bly the best ex­pla­na­tion can be found in the book “Clean Co­de”[1] by un­cle ­Bo­b, on whi­ch ex­plains that the co­de, ge­ts ou­tdated (ro­tten) wi­th the rest of the su­rroun­ding co­de, and hen­ce it makes a pla­ce for con­fu­sio­n, lea­ding to an erro­r-­pro­ne spo­t.

The­re are, ho­we­ve­r, peo­ple that seem to find so­me ar­gu­men­ts for co­m­men­ting out co­de, or lea­ving it. ­So­me co­m­mon ar­gu­men­ts/­rea­sons usua­lly are:

  • I mi­ght need this func­tio­na­li­ty la­te­r..”

We ha­ve sour­ce con­trol sys­te­ms (for exam­ple gi­t) for this. In gi­t, an­y­thing can be res­to­red from a pre­vious poin­t. If ­the so­ftwa­re is pro­per­ly un­der ver­sion con­tro­l, the­re is no rea­son to fear da­ta lo­ss. Trust gi­t, co­de fear­le­ss­l­y.

  • This is tem­po­ra­ry di­sa­ble­d… It wi­ll be res­to­red la­te­r”.

Agai­n, sa­me prin­ci­ple, re­ly on the ver­sion con­trol sys­te­m. Save a pa­tch, and then res­to­re la­te­r, or stash the chan­ges, re­vert the co­m­mi­t, etc. As you see, the­re are plen­ty of be­tter op­tions for sol­ving this sce­na­rio.

  • Co­de that was le­ft from the fist ver­sion

Pro­ba­bly de­bu­gging le­fto­ver­s. No doubt he­re: seek, lo­ca­te, des­tro­y.

The­re is, a clear pro­blem wi­th co­de that is un­der co­m­men­t, whi­ch is that it is “fro­zen” in ti­me: it was good at so­me poin­t, but then it was le­ft the­re whi­le the rest of the co­de around it, evol­ve­d, so this old co­de ­mi­ght not cer­tain­ly wo­rk (hen­ce it is “ro­tten”), so un-­co­m­men­ting it is a bad idea be­cau­se it wi­ll pro­ba­bly cras­h.

Ano­ther pro­blem is that it can be a sour­ce of bias for so­me other de­ve­lo­pe­r, who wan­ts to main­tain that co­de at a fu­tu­re poin­t in ti­me. The one who le­ft the ro­tten co­de, mi­ght ha­ve thou­ght that it was a sour­ce of ins­pi­ra­tion for when this func­tio­na­li­ty wa­s ­going to be applie­d, but ins­tea­d, it is just bia­sing the new de­ve­lo­per wi­th this ske­le­to­n, pre­ven­ting from a brand new, fres­h i­dea for that co­de.

The­re­fo­re, for the­se main rea­sons (an pro­ba­bly mu­ch mo­re), ha­ving co­de that is co­m­men­ted in a co­de ba­se, is a poor prac­ti­ce, (not to­ ­men­tion a co­de sme­ll). If you are a sea­so­ned de­ve­lo­pe­r, who ca­res about co­de qua­li­ty, and best prac­ti­ce­s, you must not doubt when ­de­le­ting it. De­le­te co­m­men­ted out co­de mer­ci­le­ss­l­y: seek, lo­ca­te and des­tro­y.

[1] A book I highly recommend: https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882

strncpy and strncat for copying strings in C

Recently, I’ve read an interesting article [1], explaining why the strncpy function in C is not safer than strcpy. The post was very interesting, but what’s more, it suggested an alternative idiom for copying strings in C, that might probably be the way to go.

La­te­r, in ano­ther arti­cle [2] that com­pa­red so­me func­tio­na­li­ty in C, Py­thon and Go, one of the co­m­men­ts ­pointed out that ve­ry sa­me idio­m. That gra­bbed my atten­tio­n, so I de­ci­ded to try it in an exam­ple.

The problem with strncpy seems to be the way it manages the source string to be copied. Based on the sample code provided in the documentation [3] (that should be just a reference), the break condition is up to n characters (the third parameter) or until the source string is exhausted, whatever happens first. This should not be a problem, unless n < strlen(source_string). That parameter would make strncpy to finish before it can put a \0 character at the end of the target string, leaving an invalid array of characters [5].

This is an exam­ple.

#include <stdio.h>  /* stdout, fprintf */
#include <string.h>  /* strncpy, strlen */

#define TARGET 10

int main(int argc, char* argv[]) {

    char *src = "Castle";  /* 6 chars long */
    char dst[TARGET] = "__________";

    dst[TARGET - 1] = '\0';

    fprintf(stdout, "%s\n", dst);   /* must be: '_________' */
    /* What happens if I pass a wrong length (lower than the actual
    * strlen) */
    strncpy(dst, src, 3);
    fprintf(stdout, "%s\n", dst);   /* must be: 'Cas______' */
    /* If I copy the string correctly, by passing the right
    * length, then strcnpy behaves as expected */
    strncpy(dst, src, strlen(src) + 1);
    fprintf(stdout, "%s\n", dst);   /* must be: 'Castle' */

    return 0;
}

On this example, the target array is represented by the variable dst, and I used a fixed-length string, on purpose for the demonstration, simulating what would actually happen. I null-terminated it so the program can finish successfully, because otherwise the operations on it would not end until the delimiter is reached, and we cannot know when that will happen, considering what’s in memory at that time. In addition, the unpredictable behaviour will lead to errors, and probably to memory corruption. The underscore, should be interpreted as slots: regions or reserved memory that are there, but empty.

The proposed idiom uses strncat (see [4]), tricking the function by passing it an empty string as the first parameter, and then the actual string we need to copy. This call will render the same result, but without the previous side effect. Let’s see an example:

#include <stdio.h>  /* stdout, fprintf */
#include <string.h>  /* strncat, strlen */

#define TARGET 10

int main(int argc, char* argv[]) {

    char *src = "Castle";  /* 6 chars long */
    char dst[TARGET] = "__________";
    dst[TARGET - 1] = '\0';

    fprintf(stdout, "%s\n", dst);   /* must be: '_________' */
    /* Prepare destination string */
    dst[0] = '\0';
    /* Copy with strncat */
    strncat(dst, src, 3);
    fprintf(stdout, "%s\n", dst);   /* must be: 'Cas' */
    /* If I copy the string correctly, by passing the right
    * length, then strcnpy behaves as expected */
    dst[0] = '\0';
    strncat(dst, src, strlen(src) + 1);
    fprintf(stdout, "%s\n", dst);   /* must be: 'Castle' */
    /* If I try to overrun the buffer */
    dst[0] = '\0';
    strncat(dst, src, strlen(src) + 10);
    fprintf(stdout, "%s\n", dst);   /* must be: 'Castle' */

    return 0;
}

He­re we see, the error is no lon­ger pre­sen­t, pro­ba­bly be­cau­se of the di­ffe­ren­ce on the im­ple­men­ta­tio­n (­the sni­ppet on the do­cu­men­ta­tion [4] gi­ves us a hint on what it does, so we can spot the chan­ge).

This might seem as a little issue, but it raised some concerns on the Linux kernel development, at the point that a new function was developed. The strscpy function is being included in the Kernel development for Linux 4.3-rc4 [6] because it is a better interface. Some of the problems mentioned in the commit message, that inspired this new version, are the ones described on the previous paragraphs.

This makes me wonder, if this should be the “correct” way for performing this operation “safely” in C. In all cases, the error is the same (not checking the boundaries, and trusting the input), and should be avoided. What I mean by this, is that we cannot simply rely on those functions being secure, the security must be in our code, so the proper way to handle these situations is to code defensively: do not trust user input, always check the boundaries, error codes, memory allocation, status of the pointer (a free for every malloc but not for a NULL pointer, etc.).

[1] https://the-flat-trantor-society.blogspot.com.ar/2012/03/no-strncpy-is-not-safer-strcpy.html.
[2] https://blog.surgut.co.uk/2015/08/go-enjoy-python3.html.
[3] strncpy documentation.
[4] (1, 2) strncat manual page.
[5] An array of characters that is not null-terminated, is invalid.
[6] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=30c44659f4a3e7e1f9f47e895591b4b40bf62671.

Setting user permissions in KVM

In a pre­vious arti­cle I men­tio­ne­d how to ins­ta­ll a li­bra­ry in Fe­do­ra, in or­der to make KVM vir­tua­li­za­tion ea­sie­r, ma­na­ging the NAT ne­two­rk con­fi­gu­ra­tion be­tween the guest vir­tua­l ­ma­chi­ne and the hos­t, by means of lib­virt.

Be­si­des tha­t, whi­le using KVM lo­ca­lly for de­ve­lo­p­men­t, I use vi­r­­t-­­ma­­na­­ger, a hel­pful appli­ca­tion that ma­na­ges the di­ffe­rent vir­tua­l ­ma­chi­nes. This appli­ca­tio­n, as we­ll as the rest of the co­m­man­ds that in­te­rac­t wi­th lib­virt (virsh for exam­ple), re­qui­re su­per user pri­vi­le­ges, so it wi­ll prompt for the su­do pa­ssword eve­ry ti­me.

This can be avoi­ded by in­clu­ding the user in­to the fo­llo­wing group­s: kvm, and lib­virt.

The­re­fo­re, just by run­ning the fo­llo­wing co­m­mand we can skip the pa­sswor­d ­prompt eve­ry ti­me.

sudo usermod -a -G kvm,libvirt mariano

This is an op­tion I would use on­ly for lo­cal de­ve­lo­p­ment on my ma­chi­ne. ­Pro­duc­ti­ve en­vi­ron­men­ts must ha­ve an strict per­mis­sions ma­na­ge­men­t.

Notas sobre la ArqConf 2015

Es­te es mi re­su­men so­bre la Ar­q­Conf 2015, la con­fe­ren­cia so­bre ar­qui­tec­tu­ra de ­so­ftwa­re que tu­vo lu­gar en la UCA el 30 de Abril de 2015. La idea es sin­te­ti­za­r ­las prin­ci­pa­les ideas que me lle­vé y re­sal­tar lo más im­por­tan­te.

Se pre­sen­tan a con­ti­nua­ción un lis­ta­do de las ideas prin­ci­pa­les por char­la con un ­bre­ve lis­ta­do de lo que más des­ta­co por ca­da uno. Nó­te­se que la lis­ta no es de ­nin­gu­na ma­ne­ra exhaus­ti­va, y ca­da sec­ción es en rea­li­dad un bre­ve pá­rra­fo ilus­tra­ti­vo­ a mo­do de re­su­men muy a al­to ni­ve­l.

Ca­da sec­ción lle­va un tí­tu­lo alu­si­vo al te­ma de la pre­sen­ta­ció­n, un bre­ve re­su­men y una lis­ta con los prin­ci­pa­les pun­tos que des­ta­co.

Orden en una arquitectura y la agilidad como atributo de calidad

Se ba­só en la ex­pe­rien­cia de un ar­qui­tec­to li­de­ran­do un equi­po de ar­qui­tec­tu­ra pa­ra un de­sa­fian­te pro­yec­to. El di­ser­tan­te ex­pli­có los pro­ble­mas a re­sol­ve­r, y el mar­co tec­no­ló­gi­co en el que se de­sa­rro­lló la so­lu­ció­n, y có­mo con una ar­qui­tec­tu­ra ele­gan­te y sim­ple con re­la­ti­va­men­te po­cos com­po­nen­tes se pue­de lle­va­r a ca­bo una im­ple­men­ta­ción de gran por­te que de so­por­te a 150.000 tran­sac­cio­nes ­con­cu­rren­tes.

  • El fac­tor cla­ve del éxi­to de una ar­qui­tec­tu­ra es la co­mu­ni­ca­ció­n.
  • La agi­li­dad de un equi­po co­mo atri­bu­to de ca­li­da­d. Es in­te­re­san­te, por­que cuan­do uno­ ­pien­sa en atri­bu­tos de ca­li­dad se le ocu­rren co­sas co­mo se­gu­ri­dad, man­te­ni­bi­li­dad, usa­bi­li­dad, es­ca­la­bi­li­dad, etc. pe­ro no el he­cho de ser ági­l. Sin em­bar­go es en ­ge­ne­ral de­sea­ble que el equi­po sea ágil y se pue­da adap­tar fá­cil­men­te a los cam­bio­s, y en ese ca­so ¿Por qué no agre­gar­lo co­mo un atri­bu­to de ca­li­da­d?
  • La fle­xi­bi­li­dad del equi­po, tam­bién co­mo atri­bu­to de ca­li­da­d. Aná­lo­go al an­te­rio­r, ­pe­ro con un de­ta­lle: si es un atri­bu­to de ca­li­da­d, tie­ne que ser me­di­ble. Lo in­te­re­san­te no es só­lo la ori­gi­na­li­dad de es­te ti­po de atri­bu­tos de ca­li­dad “no tra­di­cio­na­le­s”, ­sino tam­bién en co­mo con­si­de­rar­los den­tro de los es­ce­na­rios de ca­li­dad.
  • Los re­que­ri­mien­tos de­ben prio­ri­zar­se en el mar­co glo­bal de la or­ga­ni­za­ció­n.

Arquitectura y Big Data Analytics.

Una pre­sen­ta­ción ex­ce­len­te, con mu­cho de­ta­lle y ri­que­za téc­ni­ca, nom­bran­do tec­no­lo­gía­s, ­me­to­do­lo­gía­s, téc­ni­ca­s, y es­ti­los de ar­qui­tec­tu­ra orien­ta­dos a Big Da­ta.

  • Ka­fka co­mo he­rra­mien­ta pa­ra pro­ce­sa­mien­to de in­for­ma­ción en co­las de men­sa­jes no tra­di­cio­na­le­s (­con es­ta­do per­sis­ten­te). Es un ejem­plo de una tec­no­lo­gía que tu­vo un ca­so de éxi­to rea­l en pro­yec­tos de Big Da­ta.
  • Siem­pre guar­dar la fuen­te de da­to­s: lla­ma­da “la fuen­te de la ver­da­d” (the sour­ce of tru­th), es una bue­na prác­ti­ca, ya que tie­ne va­rias ven­ta­ja­s, co­mo por ejem­plo:
    • Pe­r­­mi­­te co­­­rre­­gir erro­­­res en ca­­so de fa­­lla (a pa­r­­tir de los da­­to­­s, se pue­­de vo­l­­ver a pro­­­ce­sar y no hay una pé­r­­di­­da irre­­cu­­pe­­ra­­ble de in­­fo­r­­ma­­ció­­n).
    • Pre­­se­r­­van­­do los da­­tos ori­­gi­­na­­les (raw da­­ta), es po­­­si­­ble en un fu­­tu­­ro ela­­bo­­­rar o ca­l­­cu­­lar nue­­va­s ­­mé­­tri­­cas si se re­­quie­­ren, co­­­sa que si por el co­n­­tra­­rio só­­­lo se gua­r­­da­­ran los da­­tos pro­­­ce­s­a­­do­­s, se­­ría i­m­­po­­­si­­ble.
    • El co­s­­to ex­­tra por el al­­ma­­ce­­na­­mien­­to no de­­be­­­ría ser un pro­­­ble­­ma, co­n­­si­­de­­ran­­do los be­­­ne­­fi­­cio­­s.
  • Pro­ce­sar la in­for­ma­ción de for­ma idem­po­ten­te: es­ta qui­zá sea la idea que me­jor re­fle­ja una ­bue­na prác­ti­ca ge­ne­ra­l, no so­lo apli­ca­ble a Big Da­ta. En lu­gar de pro­ce­sar mo­di­fi­can­do re­gis­tro­s (­por ejem­plo eje­cu­tan­do un SQL que su­me uno en al­gu­na co­lum­na), sim­ple­men­te se agre­gue una nue­va en­tra­da y lue­go el re­sul­ta­do se cal­cu­le so­bre el to­ta­l. De es­ta ma­ne­ra no se mo­di­fi­can los da­to­s, y ­de nue­vo, un po­ten­cial error es re­pa­ra­ble, no hay pér­di­da irre­ver­si­ble de in­for­ma­ció­n, etc. És­ta en ­rea­li­dad es una idea que ya exis­tía en sis­te­mas de BI, pe­ro es in­te­re­san­te no­ta­r ­que re­gis­trar los he­chos se pue­de usar pa­ra mu­chos más ca­so­s.
  • Sim­pli­fi­car las va­ria­bles tec­no­ló­gi­ca­s. En lu­gar de te­ner un ex­ten­so re­per­to­rio tec­no­ló­gi­co co­n ­mu­chas tec­no­lo­gías de pro­pó­si­to es­pe­cí­fi­co, es me­jor y más fá­cil de man­te­ner un en­torno con me­no­s ­tec­no­lo­gía­s, y, aun­que es­tas no se adap­ten per­fec­ta­men­te a ca­da pro­ble­ma en par­ti­cu­la­r, aún así hay que pri­vi­le­giar el prag­ma­tis­mo, ha­cien­do los ajus­tes ne­ce­sa­rio­s.
  • Te­ner un es­que­ma de da­tos (da­ta sche­ma) pa­ra po­der in­te­grar la in­for­ma­ción que se pro­ce­sa des­de ­di­fe­ren­tes fuen­tes.

Arquitecturas de Micro servicios.

Es muy in­te­re­san­te es­cu­char so­bre los mi­cro ser­vi­cio­s, y có­mo es­te ti­po de ar­qui­tec­tu­ras per­mi­ten una es­ca­la­bi­li­dad más fle­xi­ble.

  • Las ar­qui­tec­tu­ras de mi­cro ser­vi­cios per­mi­ten ob­te­ner la mis­ma fun­cio­na­li­da­d, pe­ro de for­ma ­dis­tri­bui­da, en contra­po­si­ción a lo que se­ría una ar­qui­tec­tu­ra mo­no­lí­ti­ca.
  • Es­to per­mi­te es­ca­lar de for­ma más fle­xi­ble, por ejem­plo se pue­den ad­mi­nis­trar los sub­sis­te­ma­s ­de for­ma in­de­pen­dien­te, asig­nan­do los re­cur­sos o man­te­nien­do más com­po­nen­tes pe­ro más sim­ple­s.
  • Es­ta se­pa­ra­ción tam­bién pue­de re­fle­jar­se en equi­pos de tra­ba­jo, áreas o pro­ce­so­s.

Arquitectura y métodos ágiles

En es­ta oca­sió­n, se ha­bló de la ar­qui­tec­tu­ra de so­ftwa­re des­de el pun­to de vis­ta de las me­to­do­lo­gía­s á­gi­les y los pro­ce­sos de de­sa­rro­llo ali­nea­dos a los re­que­ri­mien­tos fun­cio­na­les del ne­go­cio.

  • El equi­po pue­de con­ver­sar la ar­qui­tec­tu­ra en fun­ción de los re­que­ri­mien­tos con el PM, si­n ­ne­ce­sa­ria­men­te en­trar en mu­chos de­ta­lles téc­ni­co­s, con­cen­trán­do­se en la fun­cio­na­li­dad y ­com­por­ta­mien­to es­pe­ra­do.
  • És­ta con­ver­sación so­bre la ar­qui­tec­tu­ra de­be ser cons­tan­te a lo lar­go de to­do el ci­clo de de­sa­rro­llo.

Arquitectura aplicada la producción

Ex­ce­len­te cie­rre de la con­fe­ren­cia. Hi­zo mu­cho hin­ca­pié en có­mo se ve a la ar­qui­tec­tu­ra y el ro­l ­del ar­qui­tec­to o el equi­po de ar­qui­tec­tu­ra des­de el pun­to de vis­ta del CIO. És­to di­lu­ci­dó bas­tan­te ­so­bre lo que se es­pe­ra del equi­po de ar­qui­tec­tu­ra pa­ra que la or­ga­ni­za­ción fun­cio­ne.

Lo más des­ta­ca­do fue ver qué es lo que se es­pe­ra y lo que NO se es­pe­ra del ar­qui­tec­to, y có­mo­ ­lo más im­por­tan­te es po­der brin­dar una so­lu­ción co­mo in­ge­nie­ro­s, que res­pon­da a las ne­ce­si­da­des ­del ne­go­cio. La prin­ci­pal ri­que­za es­tu­vo en que las ideas fue­ron ilus­tra­das con ex­pe­rien­cias rea­le­s en Da­ta Cen­ters rea­le­s.

Al­go lla­ma­ti­vo es que mu­chas ideas men­cio­na­das son en rea­li­dad cues­tio­nes que se asu­men en un pro­yec­to­ ­de so­ftwa­re, pe­ro co­mo sa­be­mos en la prác­ti­ca no siem­pre su­ce­de, y es­to de­ri­va en ma­los re­sul­ta­do­s.

  • La in­te­gri­dad con­cep­tual es fun­da­men­ta­l: Las so­lu­cio­nes de­ben pro­por­cio­nar­se de for­ma uni­for­me, a­pli­can­do sen­dos es­ti­los y tec­no­lo­gías pa­ra los mis­mos ti­pos de pro­ble­ma­s. Aná­lo­ga­men­te, si ­pa­ra di­fe­ren­tes pro­yec­tos se usan mu­chas tec­no­lo­gías di­fe­ren­tes, el re­sul­ta­do es una ar­qui­tec­tu­ra ­gi­gan­te y muy di­fí­cil de man­te­ne­r.
  • Ca­da com­po­nen­te téc­ni­co in­terno del equi­po de in­ge­nie­ría no es el prin­ci­pal ob­je­ti­vo de la or­ga­ni­za­ció­n, ­si no que es­tán pa­ra res­pon­der a és­to­s.
  • Adop­tar nue­vas tec­no­lo­gías so­lo por que pre­sen­ta al­gu­nas ven­ta­jas par­cia­les no siem­pre es una bue­na idea a lar­go­ ­pla­zo. Sue­le su­ce­der que a lar­go pla­zo ter­mi­na te­nien­do con­se­cuen­cias per­ju­di­cia­les pa­ra el pro­yec­to.
  • Los sis­te­mas de­ben di­se­ñar­se y cons­truir­se pa­ra du­rar va­rios años (~10), y es­to im­pli­ca que la­s ­tec­no­lo­gías de cons­truc­ción tie­nen que te­ner va­rios años de exis­ti­r, de ma­ne­ra que sea ­ra­zo­na­ble ase­ve­rar que se­gui­rán es­tan­do dis­po­ni­bles el tiem­po que du­re el sis­te­ma pro­duc­ti­vo. No se­ría ­de­sea­ble te­ner que man­te­ner o ha­cer­se car­go de tec­no­lo­gías (fra­mewo­rks, toolki­ts, etc.) ob­so­le­ta­s.
  • Cri­ti­car las lla­ma­das “bue­nas prác­ti­ca­s” (o ver­da­des re­ve­la­das). Es­to sig­ni­fi­ca que cuan­do­ al­go se de­no­mi­na co­mo bue­na prác­ti­ca hay que plan­tear­se si real­men­te es así, y aun­que lo fue­ra, si esas ven­ta­jas que trae apli­can al pro­yec­to en cues­tió­n. És­ta es otra idea más ge­ne­ra­l, se tra­ta en ­de­fi­ni­ti­va de te­ner pen­sa­mien­to crí­ti­co, pe­ro es al­go que en mu­chos ca­sos no su­ce­de, y ve­mos en ge­ne­ra­l ­va­rios pro­yec­tos apli­can­do “pa­tro­nes de di­se­ño” (o de ar­qui­tec­tu­ra) o “bue­nas prác­ti­cas ági­le­s”, etc. si­n ­pen­sar real­men­te có­mo apli­can al pro­yec­to (al­go pue­de ha­ber da­do re­sul­ta­dos ex­ce­len­tes en otro pro­yec­to­, en otra em­pre­sa, en otro país, pe­ro el ar­qui­tec­to de­be con­si­de­rar si esas va­ria­bles real­men­te coin­ci­den o ­son re­le­van­tes al con­tex­to­).

>>> Conclusiones

Con­si­de­ro que la con­fe­ren­cia fue muy bue­na, te­nien­do en cuen­ta la ca­li­dad de las pre­sen­ta­cio­nes, la ex­pe­rien­cia de los di­ser­tan­tes y que to­do es­ta­ba alie­na­do con­cep­tual­men­te, lo cua­l hi­zo que la tran­si­ción en­tre te­mas tu­vie­ra una con­ti­nui­dad no­ta­ble.

Es ade­más im­por­tan­te des­ta­car que es­te ti­po de con­fe­ren­cia­s, ade­más de ser en­ri­que­cer la ex­pe­rien­cia ­pro­fe­sio­nal de to­dos (di­ser­tan­tes, or­ga­ni­za­do­res y con­cu­rren­tes), be­ne­fi­cian a la co­mu­ni­dad de ar­qui­tec­to­s.

Running RabbitMQ server on Docker

If you use Ra­bbit­MQ for de­ve­lo­p­ment fre­quen­tl­y, so­me­ti­mes you mi­ght ha­ve foun­d it uses too mu­ch re­sour­ces (i­t’s nor­mal whi­le pro­gra­m­ming to ha­ve a lot of queues or ta­sks being queued and that makes the CPU usage to spike).

Having RabbitMQ installed on the OS seems the perfect approach for production, but on development I’d rather do something different, in order to isolate the process. I know I could bound it (for example order it not to start automatically as a dæmon), by means of systemd but a few weeks ago I decided to try docker and see how it results.

It tur­ned out to be just the tool for the wo­rk, and so far wi­th a li­ttle sim­ple ­con­fi­gu­ra­tion it can run as ex­pec­te­d.

The­re is al­ready a do­cker ima­ge for Ra­bbit­M­Q, whi­ch can be au­to­ma­ti­ca­lly pu­lle­d, and then run, for exam­ple:

sudo docker pull rabbitmq
sudo docker run -d -e RABBITMQ_NODENAME=my-rabbit --cpuset="1" --name docker.rabbitmq -p 5672:5672 rabbitmq:3

The -d option indicates the process to start detached, then by passing -e we pass some environment variables (in this case, the RABBITMQ_NODENAME is a particular variable for rabbit indicating how to set the name of the node it is starting). Optionally, we can limit the CPU usage with the --cpuset, as in this case which sets the process to use the second core of the machine (it starts at 0). Then the --name is a name for the docker being created.

The most important part in this case is the port mapping, made by the -p option which in this case maps the port used by RabbitMQ directly (1:1) with the host machine. This makes the docker process to run transparently, as the other applications that try to communicate with a RabbitMQ won’t notice any difference, making it look like is executing an actual RabbitMQ service. Finally there is the name of the docker image to run.

What I usually do is to manage the docker image by its instance_id (a number that is displayed after listing the docker images, by doing sudo docker ps -a). Then we can manage it by sudo docker [start|stop] <instance_id>.

There is another command to see the output being generated by the process which is docker logs rabbitmq.docker. Notice in this case the name designated to the image was used instead of the instance_id. In addition we can see internal data for the process by running the inspect command (again we can use the instance_id or the name we assigned).

docker inspect rabbitmq.docker
sudo docker logs docker.rabbitmq

It’s important to notice that docker is actually not a virtualization platform, but a mechanism that runs processes in containers, meaning that in this case the entire RabbitMQ is running as a single process within a container, with some other limitations and bounds constrained by docker.

I found this appro­ach to be ve­ry ver­sati­le for a de­ve­lo­p­ment en­vi­ron­men­t, and wi­th Ra­bbit­MQ being the first pi­lo­t, I thi­nk I can mi­gra­te mo­re appli­ca­tions to do­cker ins­tead of ha­ving them ins­ta­lled on the sys­tem (as long as po­s­si­ble).

Find Options

Among the co­re-u­til­s, find is one of the most use­ful co­m­man­d­s. Thou­gh I use the ba­sic func­tion­s ­most of the ti­me, find has a wi­de ran­ge of pa­ra­me­ter­s, and it co­mes in handy not on­ly for fin­din­g ­fi­le­s, but al­so for ope­ra­ting a bun­ch of them at on­ce. He­re is a ve­ry sim­ple exam­ple.

Ima­gi­ne you ha­ve to mo­ve many fi­les to a di­rec­to­r­y, but they all ca­ll di­ffe­rent so a glob is no use, an­d ­ma­nua­lly mo­ving all of them is not an op­tio­n. A po­s­si­ble appro­ach would be to lo­ca­te the first of the ba­tch (for exam­ple by run­ning ls -l­r­­th). Su­ppo­se the first one of the ba­tch is ca­lled /t­m­p/­che­ck­point (for this exam­ple ­le­t’s as­su­me the fi­les re­si­de at /t­mp).

The co­m­mand would be:

find /tmp -type f -anewer /tmp/checkpoint -exec mv '{}' <target_directory> \;

The -ty­­pe f part is im­por­tant in or­der not to mo­ve the en­ti­re di­rec­to­ry (find on­ly the fi­le­s). Then we ha­ve the -a­­newer that re­cei­ves a fi­le as a pa­ra­me­te­r, and it wi­ll fil­ter for tho­se fi­les who­se ­mo­di­fi­ca­tion da­te is grea­ter than the fi­le us­ed as an exam­ple (hen­ce, this must be the start of the ba­tch), an­d ­fi­na­lly the -exec part is in­te­res­ting be­cau­se as men­tio­ned at the be­gin­nin­g, it allo­ws to per­for­m ar­bi­tra­ry ope­ra­tions on the group of fi­les (in this ca­se to mo­ve them to ano­ther lo­ca­tio­n, bu­t o­ther ac­tions su­ch as mo­di­fi­ca­tion­s, sed, etc. are al­so po­s­si­ble).

Ano­ther trait I like about find is that pre­sen­ts a se­cu­re and we­ll-­de­fi­ned in­ter­fa­ce, mea­ning that in ­so­me ca­ses I can first che­ck the re­sul­ts prior to exe­cu­te an ac­tio­n. For exam­ple, if we would like to che­ck ­for de­le­ting so­me un­ne­ce­ssary fi­le­s:

find . -name "*.pyc"

By is­suing this co­m­mand we list so­me fi­les to era­se. And then we can sim­ply do that by a­ppen­ding -de­­le­­te to the ve­ry sa­me co­m­man­d.

This is just the tip of the ice­berg of the things that are po­s­si­ble by means of the find co­m­mand an­d i­ts va­rious op­tion­s.