Fighting advent blues with magic

A bit more of a open, honest, personal round up this time: I’ve really struggled this week. Physical maladies and low vitamin d haven’t helped.

It started well, I found the root cause of a seemingly obscure problem involving compilation of c based ruby extensions, changing patch versions of ruby language and a cached folder of gems. These extensions are built against an exact version of ruby and so changing ruby patch version without rebuilding your pipeline’s cache of gems will cause an incompatible library error to be thrown, even though these changes won’t bring a change to your gemfile.lock file. 

Looking back though, this seems obvious (once you know the context) so it’s difficult to see as a real win. It’s a bit niche for the rest of the team to value either.

I’m finding it hard work to be embedded in a disengaged team. When I’m more conscientious and don’t cherry pick work items I’m left working on something no one else wants to touch and I’m not particularly inspired by either. It’s easy to feel resentful and taken for granted at that point.

I’ve got concerns about the future too, there’s a few things I’m waiting on the outcome for. They probably won’t conclude until January and I’m struggling to be patient. Apparently I get twitchy,  crotchety and destructively introspective rather than just enjoying the space to pause.

One suggested alternative is to pitch for a new role with more money. This seems decidedly counter intuitive and jaded; an opinion based on the unshakable conclusion that money is the only thanks one can ever get for a job well done. What a horrible world we live in if this is true. Time to bring on the Hygge.

In other dispiriting news, our usual annual social gathering got disrupted this year as our idea of fun didn’t match that of our friends. Their desire was to watch football in a loud crowded setting followed by a meal, four hours after my stomach thinks my throat has been cut. I need more options for social interaction methinks.

But today is magical, it is winter festival tree collection day, and it is snowing.

Cheery snowman ornament hanging from a spruce evergreen tree

Scout rules still apply

Apart from thinking ahead and finding collaborators for coaching and facilitation sessions early next year, this week has been mostly hands on coding.

I realise I miss weeks like this. There is something really satisfying in adding small features or fixing bugs and taking the opportunity to clean things up as I go along. Cleaning up can mean:

  • Checking base image versions in docker files
  • Bumping the package management version (e.g. bundler for ruby world)
  • Curating dependency chains
  • Getting the project ready for dependabot or renovate to help with dependancies
  • Considering a rename or split for a method that does more than it used to
  • DRY-ing up code where we now repeat ourselves more than twice
  • Adding a couple of tests for edge cases

It’s important to think about a few things when you decide how much to do at once. It is possible to creep all the scope and get into an update that should be its own story.

  • Do you have an automated test suite? what about build and integration pipelines?
  • How far are dependancies behind? Will we cross into breaking changes territory?
  • Is the project in sunset i.e. deprecated and mostly replaced

Doing some clean up as “small changes and often” really helps stay up to date. The longer we leave it, the more it piles up and the harder it is to face, and to tackle.

I first heard about “Code Scout Rules” or “Boy Scout Rules” in a talk given back in 2015 by Matt Cockayne and it’s still just as important as ever to leave the codebase better than you found it.

Teach someone to fish

This week I had the pleasure of facilitating an ensemble programming session for a quartet. It was their first time working in this way but they’d heard it might be worth trying. The knowledge gaps between the participants could not have been larger: highly experienced through to apprentice.

On Tuesday, in the first 30 mins they learned

  • the roles: typist, navigator, support (and facilitator)
  • the timing: 5 minute rotations and a break every full cycle
  • patience to talk to each other and really listen to each other
  • slowing down to the slowest speed was good to share knowledge

After that they wanted to try it for a full morning.

After the morning session they said they could see the real value of using ensemble working to bring everyone up to speed quicker: apprentices, new starters and late joiners to projects already in flight.

They spotted and fixed a number of bugs that had been missed before; scouts rules we naturally brought into play.

The apprentice had asked more questions, and had been able to play as navigator for the first time; this was a massive confidence boost.

On Thursday, the team retro was full of positive comments about the experience.

On Friday, one of the quartet asked me for the link to the tool I’d been using to keep track of rotations and timing. They want to run more sessions like this themselves, not all the time, but at least several a week. I couldn’t be happier.

How do you keep genuine?

This week I was asked a flattering question by a colleague during a mutual buddy chat. He asked:

when you work with junior (sic) developers in a workshop and report back a positive learning to the group, are you genuinely finding value or just supporting the learning setting?

In thinking about this I realised how much has changed in the last 3 years.

I can now admit I am rubbish at dissembling.

I can also find huge value in working with all levels via an “again for the first time” approach, an approach I use every day. It’s refreshing to meet today-colleague as a new person and today-challenge as new too.

Is this a new topic for some?

Tips for running Entity Framework (6) and MySQL / Maria

My first forays into running Entity Framework with MySQL have been entertaining to say the least. Each challenge seems to have followed the next and required a fair bit of research to solve so I thought it was worth compiling them all into one blog post. It’s an aide memoir for me and if it helps you readers out too, all the better!

bit vs tinyint for booleans

By default, EF will handle booleans as tinyint under the hood as MySQL in general has no built in concept of true | false other than 1 | 0 and uses tinyint(1) as a substitute. However, tinyint gets treated as byte rather than bit by EF so it’s all a ‘bit’ confusing (ahem, sorry)… Luckily, there is an easy code fix.

symptoms

Error messages usually seen in this case:

System.FormatException: String was not recognized as a valid Boolean..

fix

Add this to your database Context code:

modelBuilder.Properties()
            .Where(x => x.PropertyType == typeof(bool))
            .Configure(x => x.HasColumnType("bit"));

inheritance

Unlike with MSSQL, when using EF6 with MySQL, only one entity in the hierarchy can be concrete – the leaf. This can have implications for your design if you have complex similar entities and want to abstract out those things they share in common.

symptoms

Error messages usually seen in this case:

The EntitySet 'CClass' is not defined in the EntityContainer 'TestContext'

fix

  • Redesign

In a hierarchy such as
MailBox ← ExchangeMailBox ← Exchange2016MailBox
only Exchange2016MailBox can be concrete

  • Manually map inherited fields to the table in your Context via the OnModelCreating method using
modelBuilder.Entity().Map(m => m.MapInheritedProperties()).ToTable("Employee");

unique indexes

Where the design calls for a unique index on a varchar field (e.g. email address) this can be inhibited by strong internal database limits. For the unique index to work on MySQL, there is a max length that must be applied to the field. The exact value is dependent on the collation and database engine used and relates to an underlying byte value.

  • MyISAM allows keys of length 1000 bytes.
  • InnoDB allows keys of length 767 bytes.

On ascii (latin-1), characters are assumed to be stored as 1 byte, giving varchar(767) or above. However, UTF8 collations will store a char in between 1 and 3 bytes. Expect to be able to use max varchar(254).

For keys with collation utf8mb4 or UTF-16, the byte length per character is assumed as 4 bytes allowing a maximum of varchar(250) on MyISAM or varchar(190) on InnoDB.

symptoms

Error messages usually seen in this case:

Specified key was too long; max key length is 767 bytes

fix

  • consider changing your collation; does your internalisation policy require UTF-16 or only UTF-8?
  • consider changing your database engine – loads of StackOverflow articles address this InnoDB vs MyISAM debate, there’s even an article on wikipedia (to be read with the appropriate level of salt)
  • enforce the appropriate maximum length
  • enforce uniqueness programatically (and trust you don’t miss a use case in the code base)

performance issues

Although Entity Framework 6 supports the Find linq method it is not performant and leads to some very inefficient sql query creation under the hood. Big performance improvements can be made by using Single or SingleOrDefault instead.

If you have a complex hierarchy of entities it’s worth considering the question of database policy Table-Per-Heirarchy vs Table-Per-Type. There are good articles out there explaining the difference and suggesting that TPH provides better performance, particularly on Entity Framework. It makes use of a discriminator column which brings its own design  decisions. Historically there have been reports of issues with discriminator columns in earlier releases of EF6 but these appear to have been fixed.

Beyond that changing the underlying database to MSSQL or adopting a different ORM may be your only options. Looking ahead, EF7 does not yet support MySQL and no one knows when Oracle will change this so it’s worth investigating other solutions.

Lazy coding for the win

Its been a while since I wrote any new hook scripts so I thought I would refresh my knowledge before talking about them at phpconference.co.uk. I also love playing around with python – it’s straight forward, pretty and gives good ‘sad puppy’ feedback when you’ve missed a closing bracket.

So I came up with this beast to turn svn log output into a dictionary. You can then do what you like with it: output to a file, turn into html, stream to a feed, whatever.

from subprocess import check_output

repo_cmd = "svn log file:///svnrepro/tests/deleteFile/trunk"
output = check_output(repo_cmd, shell=True)

demarker = "--------------------------------------------------------\n"
contents = output.split(demarker)

ignore_these = ["[hotfix]", "[merge]"]

log_book = list()

for section in contents:
    lines = section.split("\n\n")
    if len(lines) < 2 : continue
    commit_info = lines[0]
    message = lines[1].strip()
    ignore = False
    for prefix in ignore_these:
        if message.startswith(prefix):
            ignore = True
            break
    if ignore == True: continue
    info = commit_info.split("|")
    section = dict()
    section["revision"] = info[0].strip().replace("r", "")
    section["user"] = info[1].strip()
    section["date"] = info[2].strip()
    length = info[3].strip().split(" ")
    section["messageLength"] = length[0]
    section["message"] = message
    log_book.append(section)

print log_book

What can I say, I like understanding things at a first principles level.

Of course then I remembered a cool flag which turns a predetermined text output into something much more meaningful.

repo_cmd = "svn log --xml file:///svnrepro/tests/deleteFile/trunk"

xml.etree.ElementTree is the built in python way of handling xml and, again, it’s a bit hand cranked and leads to many unnecessary lines of code. So why re-invent the wheel?

“Has someone already done this?” is rapidly becoming my first question, and yes, someone has!

https://github.com/martinblech/xmltodict

This is the second module library I tried, and is one that can actually deal with lists of elements – essential for svn log output. If you are used to handling things in JSON, it’s really easy:

from subprocess import check_output
import json
import xmltodict
# https://github.com/martinblech/xmltodict

repo_cmd = "svn log --xml file:///svnrepro/tests/deleteFile/trunk"

output = check_output(repo_cmd, shell=True)

print json.dumps(xmltodict.parse(output))

I have to admit, a little piece of me feels like I’m no longer writing ‘real’ code and that there’s nothing left unsolved or uncoded. The rest of me is doing a happy dance – now we are freed up to use technology to solve the real issues – save time, collaborate well and communicate effectively to make lives better!

Keeping my promise

This was very nearly about breaking my promise, in fact I had drafted a blog post full of excuses as to how my year had been miserable and there was no way I could fulfil my promise of February. Then I panicked, ditched the post and emailed the wonderful and supportive Jenny (@miss_jwo) confessing my failure.

A little while later something wonderful happened, I spotted an opportunity and careless of the consequences, grabbed it with both hands. I surprised myself. A few emails later and phpsw user group had kindly accepted my proposal to give a lightening talk. The talk was last Wednesday night. Their warm welcome was a real testament to the friendly community that is a php user group. I am happy to report I felt calmer than I have ever felt in front of 40+ strangers and I am looking forward to doing it again. So much so in fact, I am considering submitting to other user group nights and even conferences!

Along the way I have learned a few great things about speaking:

  • it’s really easy to arrange
  • it doesn’t have to be a new idea, just your take on it
  • audiences come up with the coolest questions (and tips!) to make you think
  • turning up early wins you brownie points and more free drinks choices
  • practicing your talk to yourself in a spare office the week before is worth it just for the funny looks you get
  • lots of user groups are looking for new speakers of any experience level
  • you might even get your petrol money back

Most of all I got to keep a promise and that’s worth everything.

No shoddy work in the free world

When evaluating a feature for value it’s easy to fall into the trap of equating value with money; direct revenue is not the only value you can get from a feature. Here are some other concepts of value:

  • ease of maintenance, monitoring and admin tools to quickly identify and diagnose potential issues (keep good customers)
  • framework designs can save time adding new features and allow parts of your system to be resold as services (robust resellable software)
  • the right free features can draw customers to your product in the face of stiff competition, gaining you greater customer numbers (get more market share)

The benefit from these features is completely wasted if their execution is ill thought out, is flaky or worse, they don’t work at all.

If you hear that someone isn’t ‘allowed’ to put too much time into a feature because it’s offered for free or its only internal, alarm bells should be ringing. The same key aspects of software should apply here as for anything else:

how this feature should work for each and every user type

Does every type of customer interact with the feature in the same way? Are there staff only actions? Does the feature work in different ways on different tariffs?

how it should be presented and supported

How does the interface appear? Does the ‘flow’ of the feature make sense to the intended audience? What help is available immediately alongside the feature? Is it easy to get to more detailed help? How do support staff learn about this feature?

what design will allow it to be flexible for the future

Does this feature scale appropriately? How much does it share with other features? How often do they change? Do we anticipate offering this through another channel or interface? Is there the possibility of a new type of customer we could be offering this to?

who we should be collaborating with

internal users, infrastructure experts, trainers, support staff, beta testers, external suppliers, industry partners

how we can develop it to be easily maintained and monitored

active monitoring and analysis tools for predictive performance, usage and capacity; intelligent logging and log file analysis; alternative access to feature data including admin tools; api endpoint test harnesses and tools

how we should be testing

when we should be testing; who should be testing; what we are testing for: functionality, usability, performance, load

the most efficient, and least disruptive, release process and cycle

where we deploy, redundancy and failover, how often we release, minimising outages, maintaining reliable and predictable service.

If your free service is mentioned as a big selling point in all the marketing material and brings in twice as many customers, you’ll be glad to have shunned the shoddy and invested wisely in the free world.

How to write yourself positive

What you write becomes the historic record of what happened, if that record is full of negativity, the memory becomes negative.

Apparently, when you remember something, you are actually recalling the last time you recalled it, not the memory itself!

So how do you help your future self out?

Write everything down sentence by sentence. read back the sentence and chop into small statements. more than one and gives you a dividing line. ‘but’, ‘then’, ‘so’ are also clues. Ask yourself do all the parts need to be there? if your record should be factual, without emotion, cut parts that are interpretation, hearsay or feelings. If you are challenging the negative, cut out anything grumpy, limiting or jealous. leave in achievements, benefits and lessons learned.

What you are left with can make you feel uncomfortable right now. This record can challenge how you see yourself and what happens in your life. That’s not a bad thing if you embrace if for the learning experience it presents.

Your edit might also seem terse or disjointed. On the plus side it will be more quickly read and understood by others. Your future self, friends and colleagues will thank you for the time saved. All the more effort available for your next positive experience!

Out from behind the mic stand

Yesterday was the first band practice I took the mic out of the stand and sang with just the mic. It was a revelation. Eventually.

First it felt very scary and exposed. Then I realised I didn’t have to stand up straight, I didn’t have to hold my arms up across my ample busom. I could move around!

Movement is expression, I started smiling, moving my one arm out to my side, opening my chest, lifting my ribs. I could suddenly get more air and that high note I’d been drifting around for ages? I belted it out, I nailed it and the other folks in the band? They noticed (and complimented me, thanks guys 🙂 )

Hiding or propping myself up is easy and safe. It’s something we frequently do to make things ‘quicker’ or easier on ourselves. Props can be tools, practises, services and often people. Blindly relying on dodgy dropping if stands, unmaintained tools, ill fitting practises, out of date services is not going to help progress in the short term or the long run. Assuming that an expert in one field knows absolutely everything, is completely up to date and is senior in every field is darn risky to say the very least.

Not hiding ourselves is scary to start with, we are exposed and seem vulnerable: a bit like butterflies just hatching. But. Not hiding gives us so much more room to play, to experiment, presents us with the freedom to be the biggest and best form of ourselves. We can give so much more, share what we really know and understand.

Coming out from behind the mic stand is much easier than spending hours trying hit a note that a restricted set of lungs will never be able to produce.

So what are you hiding behind that holds you back?