Hoping the Brexit vote was a protest vote, not a racist one

The only positive I hope from this is that the Brexit vote was a protest vote.

So I’m hoping the majority of ‘leavers’ aren’t siding with Farage, they don’t care about Boris – they just want to say Fuck You to all politicians and leaders.

This is probably the first and only time that the entire population have been given a protest vote. This is not a choice between one dodgy politician or another politician. This is not choosing between Trump or Clinton.

This is a chance to say fuck you to every single party. This is fuck you to the leaders of both the Conservative and Labour parties.

When this chance comes along – you don’t care about the consequences. You don’t care if the ‘leave’ campaign is spouting lies, you don’t care about the doom spouted by the ‘remain’ campaign. You just want a chance to say fuck you. You know the system is wrong and this is the only tool you’ve got to say so.

I certainly respect the people who protest for what they believe in. Perhaps I’m wrong for thinking all the pensioners don’t care about the consequences. Perhaps they know better than me that the pain of leaving is worth it.

There are some heart felt comments from leavers, not based on the crap spouted by the leave campaign, in this blog post Dear Brexiteer. What we need you to do now.:

“I voted leave , There its out there .. I trusted DC to come back from Brussels with a list of pledges that would let us have some tools to work with to make me feel that improvements could be made to the way we live..
Be it a better NHS ,Schools ,social care services,security. ..
He came back with nothing and nothing was offered .. in fact we were told that it would never change..
So people who have had enough like myself and 17 million others voted with our feet in the only way we know .. a very British revolt ..
Now we are being called racist and xenophobic but this is just so untrue for the masses.. we just need change .
The EU is a broken antique of a monster that isn’t up to listening to the working classes..
We have been called inward looking but again that is not true. . I concider myself Global I want to be able to talk with anyone across the world’s economy. .
Things will never be the same again and for that I feel that my cross mattered.
The first time I think the working class has ever mattered…”

So I hope when all the dust settles that we’re all still willing to tell Farage and Johnson to go fuck themselves (I’m pretty sure not many people care about what Gove says).

Functional Learnings of Python

Photo credit: cheers Hembo Pagi

This is my attempt at converting a solution to a pascal’s triangle problem that I’d written in an imperitive manner into a functional one. I kind of hope that anyone reading this is also trying to figure out the meanings behind functional programming, I’m trying to describe all the steps that I go through.

It is a mini way of me trying to discover what being ‘declarative’ actually means.

I know the kind of definition e.g.:

  1. “say what you want” not “how to do it”
  2. picture vs recipie
  3. sqrt(2) vs looping from x = 1 finding the mean of x and 2/x
  4. SQL, Haskell vs C++, Java

This is what I understand about functional programming:

  1. no side-effects
  2. focus on the functions (functions are first class objects)

The imperative

Here was my imperative method to return the k-th row of Pascal’s triangle. Now I’ll be honest and say this took me hours to get a correct result when I first tried to implement it in an imperative manner.

class Solution:
    # @param A : integer
    # @return a list of integers
    def getRow(self, A):
        pascal = [1]

        for i in range(A):
            pascal.append(0)
            for j in range(i,0,-1):
                pascal[j] += pascal[j-1]

        return pascal

How do I say that declaratively?

It is supposed to be something like (see Functional Programming in Python):

for thing in collection:
    process(thing)

Although I don’t really understand how that would be declarative. What’s my collection?

I kind of understand how SQL is declarative

SELECT  field
FROM    table
WHERE   other_field > 5

Instead of doing one big for loop.

What I don’t understand is how you can loop over an array looking at the current and previous elements.

To loop in a functional manner you need to be able to treat each element independently.

What I spotted was that you can copy and shift the array by one. So:

#             current row = [1,3,3,1]
#      append zero to row = [1,3,3,1,0] (1)
#     prepend zero to row = [0,1,3,3,1] (2)
# sum elements of 1 and 2 = [1,4,6,4,1]

This seems kind of neat, but I don’t really know how inefficient this is.

I also don’t know if that’s very functional. But I’m pretty sure that you can then loop over the elements of the arrays and sum them in a list comprehension, which always seems to be ‘functional’ (it’s final all encompassing functional solution suggested by this IBM article on Functional programming in Python).

The best I have so far is, for ‘what’ we want

for i in range(k) # k-th row
    current_row = get_next_row(current_row)

But I couldn’t quite figure that out.

The next point that seems to be ‘ok’ with functional programming is recursion, which seems to be a sensible way of accessing current and next/previous

def get_next_row(row, i):
    if i = 0:
        return row
    return get_next_row(calc_new_row, i-1)

Haskell

At this point I turned to Stack Overflow. What I learned is that trying to search for functional programming solutions in Python isn’t very rewarding. Most people write imperitively in Python so you get a lot of chaff in your search results.

What made more sense was searching for Haskell solutions. The second solution I came across was Making a list of lists to compute Pascal’s triangle in Haskell, with this solution:

nextRow vs = [1] ++ zipWith (+) vs (tail vs) ++ [1]
allPascals = iterate nextRow [1]

Now you don’t have to know a ton of Haskell to understand the code. Also it helps if you know the zip python syntax which is similar, for summing the elements of two arrays.

[x + y for x, y in zip(first, second)]

Also this is following my concept of summing one array and itself shifted (by taking the ‘tail’):

# [1,3,3,1]
# [3,3,1]
# [4,6,4]

This gives you everything except the outer [1] elements, which get concatenated using the Haskell ++ operator (just + in Python). N.B. this also works because zip only processes until the end of the shortest list.

The first attempt

So I combined this all into the following recursive function:

def next_row(row, i):
    if i == 0:
        return row
    new_row = [1] + [x + y for x, y in zip(row, row[1:])] + [1]
    return next_row(new_row, i-1)

The great thing was that this code was almost completely perfect at solving the problem the very first time I wrote it. The only problem I had was that I had mistyped the ‘tail’ and written row[:1] instead of row[1:]. Once I fixed that the code worked perfectly. Plus that was an easy, 5 minute bug to fix.

This is actually really, really exciting for me. To have code that works (passed the tests (correct results time and) and was accepted as a submission) first time off is awesome. The big time drain that I hit when solving coding problems was if your solution doesn’t work and having to bug fix things. This is what turns a 30 minute problem into an 4-10 hour problem.

The improved attempt

N.B. Anyone learning from this there are definite issues with this section – I’m mutating row. See the last section for more details. I’m leaving this here though to show the ugly backward steps I made.

I still wanted to try and improve this though. I think a ‘smell’ in functional programming is if you end up with an if statement in your code.

So I made a futher attempt by introducing a lambda:

class Solution:
    def getRow(self, A):
        process = lambda row: [1] + [x + y for x, y in zip(row, row[1:])] + [1]
        row = [1]
        for i in range(A):
            row = process(row)
        return row

This code again, worked perfectly. It would have suffered from the same ‘tail’ bug I had previously, but prevents the possibility of bug in the base case of the recursion.

This then becomes more similar to the Haskell:

nextRow vs = [1] ++ zipWith (+) vs (tail vs) ++ [1]
allPascals = iterate nextRow [1]

Conclusion

I still don’t really know if this is more ‘declarative’.

I’ve replaced

for i in range(A):
    pascal.append(0)
    for j in range(i,0,-1):
        pascal[j] += pascal[j-1]

with

process = lambda row: [1] + [x + y for x, y in zip(row, row[1:])] + [1]
for i in range(A):
    row = process(row)

It certainly is more similar to what the book suggested was ‘declarative’. But I could have just sub-functioned the inner for loop of the imperitive solution.

What it does do is eliminate the middle for loop and reduce the various i,0,j-1,-1 elements which cause the bugs and are hard to track down.

In the functional code the only similar, ‘off-by-one’ style code is for row[1:] which is where my only bug appeared.

It is also useful that it effectively encourages you to create sub-functions.

So I still don’t know what declarative really is, but the code I seem to write searching for it seems more robust.

Update

row = process(row) is not functional. It’s mutating row. I realised this, but as part of my learning I couldn’t think of a functional way of fixing it. Watching an OSCON video on Functional Thinking, I think that I actually need ‘reduce’ steadily build up the pascal row. This is work in progress…

Jupyter on Windows with Chocolatey

Install Python

This is a quick post (and will probably be outdated quickly), but it took me a while to get it correct.

I wanted to install Jupyter on Windows. The recommeded route is via Anaconda but I like to use Chocolatey the Windows package manager as much as possible.

I used Chocolatey to install Python package (currently v3.5.1). Assuming you’ve installed Chocolatey already, run as administrator:

choco install -y python

This installs python to C:\Program Files\Python35. It also installs and pip.exe to C:\Program Files\Python35\Scripts.

Add C:\Program Files\Python35 and C:\Program Files\Python35\Scripts to your PATH then you should be able to run python and pip immediately.

Install PIP (hint: it’s already installed)

Note that pip gets automatically installed when you install python – nothing extra is required.

You can get led astray here by an official guide which suggests having to download and run ez_setup.py.

Don’t do this.

There is also confusingly a Chocolatey PIP package. Again don’t install this.

Otherwise you might end up here.

Running PIP packages

Because we’re using Chocolatey, the installation doesn’t follow the recommended installation for Jupyter.

The main thing is that you need to add the python Scripts directory to your PATH, as above.

Install Jupyter

Run as administrator:

pip install jupyter

This will install a jupyter.exe file into the Scripts directory. If that directory is in your PATH, then you should be able to directly run jupyter commands.

Create a Jupyter notebook

Jupyter runs inside the browser but installs files and all the save data in the directory that you run the jupyter command from.

mkdir jupyter
cd jupyter
jupyter notepad

If everything is working that will open up your browser at http://localhost:8888/.

Getting the gist of segment trees

Segment Trees

Wow they seem overly complex to learn about. They’re a very easy concept, but there is a paucity of resources for explaining them.

Have a gander at the resources at the bottom for the ones I came across. Took me ages to figure out the differences between them.

Here, I assume you know about Binary Trees and recursion on Binary Trees.

Any code is in python which is about as close to pseudo code as you can get.

tl;dr A segmented tree in one diagram

Structuring an array (length 8) in a segmented tree. The indexes of the array are the leaf nodes.

The parent nodes here are sums of the children (but could be another function).

This can be coded as a normal binary tree.

       28
     /   \
   6       22
  / \     / \
 1   5   9   13
/ \ / \ / \ / \
0 1 2 3 4 5 6 7

Step 1: Create a tree with specific indexing strategy

We are going to use an example for storing 2^3 = 8 values indexed by [0,7] from an array in a segment tree.

We’re using 8 to ensure the segment tree we will create is a “perfect” binary tree to make things easier, or at least prettier. All segmented trees are ‘full‘ – all nodes have 0 or 2 children.

The values will get added as leaves of the tree.

So we want a tree that has 8 leaves.

To learn about how we construct them we’ll first number the nodes of a tree in depth wise (across the levels).

Some (Geeks for Geeks) start this at 0. League of Programmers start this at 1.

A perfect binary tree with 8 leaves has 4 levels and 2^4 – 1 = 15 nodes. The eight leaves are the values so we will use the 15 – 8 = 7 nodes for the segment nodes.

A binary tree with 7 nodes indexed by level, starting at 1 and with the 8 values of our array as the leaves:

       1 <= parent
     /   \
   2       3 <= i
  / \     / \
 4   5   6   7 <= children
/ \ / \ / \ / \
0 1 2 3 4 5 6 7

For each node i that we have indexed, e.g node i = 3, then

     parent = i/2 = 1 (integer division)
 left child = 2*i = 6
right child = 2*i+1 = 7

Alternatively, If you index the nodes from 0:

       0 <= parent
     /   \
   1       2 <= i
  / \     / \
 3   4   5   6 <= children
/ \ / \ / \ / \
0 1 2 3 4 5 6 7

Here for i = 2:

parent = (i-1)/2 (integer division)
  left = 2*i + 1
 right = 2*i + 2

The difference between starting from 0 or 1 confused me a lot when I was looking at the various tutorials. I’m going to stick with indexing from 1.

Step 2: Define what the segments are for each of the nodes.

Reminder about ranges:
() = ‘open’ boundary, [] = ‘closed’ boundary. We’re only using closed boundaries here.

x in (a,b] => a < x <= b
x in [a,b) => a <= x < b
N.B. [a,a] = a

The root node i = 1 refers to the whole array range [0,7]. Then node i = 2 refers to the lower half [0,3] and node i = 3 refers to the upper half [4,7]

N.B. [0,7] is the range 0-7 but it can also refer to a python list with 0 and 7 as values. Hopefully the context should make it obvious.

So, so far we have:

       1
     [0,7]
     /   \
   2       3
 [0,3]   [4,7]

Then replacing all indexes of the tree with their corresponding segments:

         [0,7]
        /     \
   [0,3]       [4,7]
   /   \       /   \
[0,1] [2,3]  [4,5] [6,7]
 / \   / \    / \   / \
 0 1   2 3    4 5   6 7

Step 3: Create a function to store the segment of values

We want to set the value of the parent nodes to indicate the range of values that are in the leaves of the current sub-tree. This is quite well explained on Slide 16 of the League of Programmers’ slides:

“Each internal node represents some merging of the leaf nodes”

There are four common ones: sum, min, max, GCD (Greatest Common Divisor).

In more general terms, for node i = 3, node value = f(a[4], a[5], a[6], a[7]) for the array a.

However this function f can be anything, e.g. other possibilities are count or finding the mid point of the range ((max - min) / 2)

Min is used for a the the Range Minimum Query, Geeks for Geeks have a Coded Example.

Example using sum:

       28 (= 6+22)
     /   \
   6       22 (= 9+13)
  / \     / \
 1   5   9   13 (= 6+7)
/ \ / \ / \ / \
0 1 2 3 4 5 6 7

This corresponds to the similar tree in the Geeks for Geeks tutorial.

Step 4: Worked example to ‘fill in the blanks’

We’re going to use count as the function. We want to fill in the ith blank position (starting from 1).

We are given an array of [3,2,1,1] for which blank to fill for the following sorted values [1,2,3,4].

    blanks = [3,2,1,1]
    #         |
    #         ^
    values = [1,2,3,4]

So 1 wants to go into the 3rd blank space. Then 2 wants to go into the 2nd blank space of the remaining 3 spaces.

Step 1: [_,_,1,_] <= 1 goes in 3rd blank space
Step 2: [_,2,1,_] <= 2 goes in 2nd blank space remaining
Step 3: [3,2,1,_] <= 3 goes in 1st blank space remaining
Step 4: [3,2,1,4] <= 4 goes in 1st blank space remaining

       0 <= count of values in segment [1,4]
     /   \
   0       0
  / \     / \
 1   2   3   4
 _   _   _   _

Fill in 1:

       1* <= count increased by 1
     /   \
   0       1*  <= count increased by 1
  / \     / \
 1   2   3   4
 _   _   1*  _ <= 1 placed in 3rd blank

Fill in 2:

       2*
     /   \
   1*      1
  / \     / \
 1   2   3   4
 _   2*  1   _

Fill in 3:

       3*
     /   \
   2*      1
  / \     / \
 1   2   3   4
 3*  2   1   _

Fill in 4:

       4*
     /   \
   2      2*
  / \     / \
 1   2   3   4
 3   2   1   4*

Step 5: Code for ‘fill in the blanks’

We’re assuming that the tree has been created for us. The code for this is at the end.

To programmatically decide which blank to fill, we need to know if the ith value is in the left or right segment. Here’s the starting point again.

Initially, the counts are all zero. So for the root node we want to know if 3 is greater than or less than 4/2 (half the length of the array). If it’s equal, i.e. 2, then we want to put it in the left segment. Also if it’s less than i.e. 1 put it in the left half. If its greater than i.e. [3,4] then it wants to go in the right segment.

    N = len(segment)
    if N/2 < i:
        # go right
    else: # i <= N/2
        # go left
       0 <= looking for 3rd leaf node, i = 3 > N/2 so look right
     /   \
   0       0
  / \     / \
 1   2   3   4
 _   _   _   _

At the second level, the count is still zero. But now we are only considering half the values. so N = 2, but now we want to know if the initial i = 3 goes in the 1st or 2nd segment. We’ve discarded half the values so we can subtract half the original N = 4 values from i. So at the first level of recursion, we alter i for the next level to be, i = i - N/2. Going left requires no change.

The length of the segments, if even, will divide in two, so N = N/2. If odd, then N/2 (using integer division) go left and the rest (N - N/2) go right so we split range [1,N] into [1,N/2] and [N/2+1,N]. So [1,3] => 1 and [2,3]

       0
     /   \
   0       0 <= i = 1, N = 2, i <= N/2 so look left
  / \     / \
 1   2   3   4
 _   _   _   _
    def fillBlank(self, segment, i, N):
        # N = len(segment) # we pass this in
        if i <= N/2:
            # go left
            N = N/2
            self.fillBlank(segment.left, i, N)
        else: # N/2 < i
            # go right
            i = i - N/2
            N = N - N/2
            self.fillBlank(segment.right, i, N)

Then when we hit the leaf then we can set the value.

    # N.B. the new params blanks and value
    # blanks is passed in by reference
    # value is going to fill the blank
    def fillBlank(self, segment, i, N, blanks, value):
        if not segment.left and not segment.right:
            # N == 1
            # we're dealing with array indexes, so minus 1
            blanks[segment.value - 1] = value
            return
        # ...
       0
     /   \
   0       0 <= i = 1, N = 2, i <= N/2 so look left
  / \     / \
 1   2   3   4
 _   _   1   _ <= blanks[2] = 1

Now, to complete our first pass through the tree, we should have updated the counts (the values of each of the nodes) by one as we passed through it. So the full method becomes:

    def fillBlank(self, segment, i, N, blanks, value):
        if not segment.left and not segment.right:
            # N == 1
            # we're dealing with array indexes, so minus 1
            blanks[segment.value - 1] = value
            return
        # increment the count of this segment
        segment.value += 1
        # N = len(segment) # we pass this in
        if i <= N/2:
            # go left
            N = N/2
            self.fillBlank(segment.left, i, N, blanks, value)
        else: # N/2 < i
            # go right
            i = i - N/2
            N = N - N/2
            self.fillBlank(segment.right, i, N, blanks, value)

However this doesn’t quite work for all the following passes through the tree, e.g.searching for the 1st blank space for the very final step – which wants to go into the blank space which is at the end of the array.

To do this we take into account the count of the child segments. If the left segment has been filled up but the right segment is empty, then we automatically want to look in the right segment.

So the condition i <= N/2 becomes i <= N/2 - X where X = segment.left.value and when going right, we alter the ith element similarly i = i - (N/2 - X).

Finally we don’t care about this conditional check when the segment only has two values in it as those two values are just the leaves (which don’t contain counts). This works since all segmented trees are ‘full’ (every node has 0 or 2 children).

    def fillBlank(self, segment, i, N, blanks, value):
        if not segment.left and not segment.right:
            # N == 1
            # we're dealing with array indexes, so minus 1
            blanks[segment.value - 1] = value
            return
        # increment the count of this segment
        segment.value += 1
        # N = len(segment) # we pass this in
        # this section gets more complicated from edge cases
        if N > 3:
            X = segment.left.val
        elif N == 3:
            # left node is leaf
            X = segment.val - segment.right.val
        elif N == 2 and blanks[segment.left.val - 1]:
            # need to check if left has been filled
            X = 1
        else:
            X = 0

        if i <= N/2 - X: # X subtracted
            # go left
            N = N/2
            self.fillBlank(segment.left, i, N, blanks, value)
        else: # N/2 < i
            # go right
            i = i - (N/2 - X)
            N = N - N/2
            self.fillBlank(segment.right, i, N, blanks, value)

Here’s the first step of the second pass:

       1 <= looking for 2nd leaf node, X=0, N=4, i=2 <= N/2 so look left
     /   \
   0       1
  / \     / \
 1   2   3   4
 _   _   1   _

We then follow the recursion through, which should populate the blanks list.

Resources

view raw
segment-trees.md
hosted with ❤ by GitHub

Lock Stock Pomodoros

I give you http://pom.ianchanning.com. Pomodoros with a Lock Stock film quote at the end of each one.

I’m a little obsessive about timers (see bottom) and the other pomodoro timers I’ve tried failed on different counts.

I’ve tried to cram everything I find useful about a timer.

Firstly spacebar to start/stop. The timer should be something that sits in the background and requires minimal attention. So a quick ctrl+tab to get the the pomodoro tab and then hit space and then back to your work.

Secondly the timer appears in the title bar and it appears first. Then you can easily see that it’s running so that you can still see the timer when you have lots of tabs open.

Thirdly there’s a start and stop. Possibly this flies against what a Pomodoro timer should do. However any timer I have I want to be able to start/stop/reset. That’s just what a timer should do, it’s annoying if it doesn’t.

Fourthly the timer has to notify you properly when the pomodoro finishes. This is achieved through a buzzer and a browser notification. Notifications sometimes don’t work or you don’t see them so you need the buzzer backup. Sorry IE folks your browser is still terrible and can’t handle the Javascript Notifications and I can’t be bothered to fix it when it’s just for a personal project.

Fifthly a count of the pomodoros is useful to know how many you cranked out in the day.

Sixthly, a bit of browser responsiveness goes a long way.

And that’s it. No more please.

Well except I needed to put something in the notification so I jazzed things up with quotes from Lock Stock and Two Smoking Barrels. If you can’t take the ‘c’ word at least once then this timer is not for you.

For those who got this far. I started with the most basic timer I could think of that still had a simple start-stop mechanism. Then after staring at the jquery code I’d copied I realised I could make a generic jquery ticker function. So I can tick 25 minutes down and then tick the pomodoro count up 1 each time. The rest is a codged together front end on top of that.

Linux Mint Dual Boot

It’s not supposed to be this painful.

I finally (9 months after first attempt) got a UEFI dual boot working.

TL;DR Follow one of these:

  1. http://itsfoss.com/install-ubuntu-1404-dual-boot-mode-windows-8-81-uefi/
  2. http://www.tweaking4all.com/os-tips-and-tricks/uefi-dual-boot-windows81-ubuntu/

I’m calling Linux Mint my lucky charm as that was the first one I got working as a Vagrant desktop and it’s worked perfectly now that I’ve sorted out the other issues.

Having read and re-read so many articles – there were three final things I had problems with/didn’t understand:

  1. Unetbootin was creating blank USB drives fixed by using universal-usb-installer as recommended by both articles above
  2. UEFI bootable USB stick Universal USB works fine (you can create a UEFI only one by simply extracting an ISO using 7-zip and just copying the files, which ignores the extra step that also installs a MBR file (I think))
  3. Booting to USB when the BIOS is set to UEFI In Windows hold shift and click ‘Restart’ (fucking magic) – this point is buried as almost an aside in the main itsfoss.com article above:

To boot from USB, will have to choose boot from USB option from within Windows itself. Either with PC Setting (like for UEFI) or pressing shift key while clicking on Restart.

Point 3 is also in the wonderfully in-depth tweaking4all.com article.

The rest seemed quite ok as I had the rest of the BIOS settings / fast boot options correct.

Once I got the USB stick working, I ran the check disk once and then went into the live Linux Mint. The first happy thing was that the Wifi was working straight off – which wasn’t before with Ubuntu. I then followed the tweaking4all guide to create:

  • first a root (/) /ext4 logical partition
  • leaving 8GB (8191MB, to equal the RAM size) of space for the swap logical partition.

After Linux Mint installed – basically everything was perfect. Where as the tweaking4all guide talks about running the ‘Boot Repair’ program from Mint before restarting. However I took a gamble based on the itsfoss article that a reboot should work. And it did.

The GRUB bootloader was happily installed into the EFI ‘partition’ (not sure if that’s the right word) so that now starting the computer shows the dual-boot screen.

Then booting Linux Mint works perfectly fine – and again the Wifi was working. As was the other major obvious one of the power button working.

As a minor anecdotal point – the booting times (Lenovo Yoga 13, Core i7, 8GB RAM, SSD) for Mint vs Windows was 20s for both to get to the desktop after logging in. There goes my theory of how gloriously fast Linux is to load. I guess that just isn’t the case with Ubuntu based systems.

Now to blat Mint and see if Fedora is any faster.

Scrum unsucked

[Photo credit: darkmatter]

I’ve been reading into Scrum closer recently and it just seems like a horrible corruption of an inspirational idea, that could much more easily be integrated into normal project practices.

Agile

I love the agile principles. I’m repeating them here for my own benefit, I like a good list as much as the next.

  1. Customer satisfaction by rapid delivery of useful software
  2. Welcome changing requirements, even late in development
  3. Working software is delivered frequently (weeks rather than months)
  4. Working software is the principal measure of progress
  5. Sustainable development, able to maintain a constant pace
  6. Close, daily cooperation between business people and developers
  7. Face-to-face conversation is the best form of communication (co-location)
  8. Projects are built around motivated individuals, who should be trusted
  9. Continuous attention to technical excellence and good design
  10. Simplicity—the art of maximizing the amount of work not done—is essential
  11. Self-organizing teams
  12. Regular adaptation to changing circumstances

Each principle makes sense, even if, perhaps, it isn’t attainable.

Their manifesto is harder to figure out, but still has a great aspirational feel to it.

We value this Over this
Individuals and interactions Processes and tools
Working software Comprehensive documentation
Customer collaboration Contract negotiation
Responding to change Following a plan

However importantly note that the second column is still valued, but just not as much as the first.

Scrum

Now read the Scrum Guide™ (TM!). Scrum just seems like a conversion of agile concepts into sucky word project management.

I realised I’d already got it wrong with what a sprint was – I thought it was two weeks work. They suggest that it should be a month’s worth of work – which makes sense, but why not just call it a month’s worth of work – or if it’s going to be less just call it an increment.

Anyway I read through the Scrum Guide and here’s my unsucked version of their main concepts.

Scrum Guide™ Unsucked agile guidelines™
The Scrum Team The Team
The Product Owner The Client
The Development Team The Developers
The Scrum Master The Project Manager
The Sprint The Sprint (it isn’t actually that bad a word) / Increment
Sprint Planning Planning
Daily Scrum Talk
Sprint Review Sprint Update
Sprint Retrospective Sprint Review
Scrum Artifacts Documents
Product Backlog Issues
Sprint Backlog Sprint Issues
Increment Completed Issues
Artifact Transparency Simplification
Definition of “Done” Happy Client

Can I have a word…

Scrum style

“Shall we go over the Sprint Review later? I’m not convinced that the Product Backlog contains all that we require for the definition of done. The Daily Scrum this morning wasn’t great as the Scrum Master had to get the Development Team to better explain their progress to The Product Owner. We need to make the Sprint Backlog artifact more transparent as The Product Owner feels we haven’t followed what we agreed at the last Sprint Retrospective.”

vs sane

“Shall we go over the Sprint update later? I’m not convinced that the issues contain all that we require for the client to be happy. The talk this morning wasn’t great as the Project Manager had to get the developers to better explain their progress to the business. We need to simplify the Sprint’s issues as the client feels we haven’t followed what we agreed at the last Sprint review.”

Please let the first type of conversation cease.

New Relic monitoring for Server, Virtual Host PHP and WordPress

New Relic offers completely free server monitoring (CPU, RAM, Network I/O etc) and limited (24 hour retention) PHP application performance monitoring (detailed error logs, PHP vs MySQL load times etc). Its far better than monitoring tools I’ve seen and very useful for trying to figure out any problems with WordPress which can be quite heavy in different areas CPU vs RAM vs Database.

Server monitoring

New Relic have excellent docs on the server installation, but this is a summary of the commands, N.B. the YOUR_LICENCE_KEY needs to be changed:

sudo sh -c 'echo deb http://apt.newrelic.com/debian/ newrelic non-free > /etc/apt/sources.list.d/newrelic.list'
wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install newrelic-sysmond
nrsysmond-config --set license_key=YOUR_LICENCE_KEY
/etc/init.d/newrelic-sysmond start

PHP monitoring

There’s again useful New Relic installation notes:

sudo sh -c 'echo deb http://apt.newrelic.com/debian/ newrelic non-free > /etc/apt/sources.list.d/newrelic.list'
wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -
sudo apt-get update

This is all repeated from the Server section, so the only actual line required is:

sudo apt-get install newrelic-php5

This asks you to give an Application Name – the best format to use is [vhost domain];[server domain] where the server domain is used as a kind of parent filter for all vhost domains.

PHP vhosts applications

This doesn’t quite work for Plesk installations as we want a per-directory application performance indicator. This assumes that we’re using the Apache php mod rather than FastCGI as we’re putting PHP ini settings in apache config files.

Plesk

This guide (http://blogs.reliablepenguin.com/2015/02/11/plesk-new-relic) was the perfect starting point. The application names they suggest are really useful e.g.

[vhost domain];[server domain]

Changing the application name

If you make a mistake with the application name you can change it. Just follow this document (https://docs.newrelic.com/docs/apm/new-relic-apm/maintenance/renaming-applications).

Apache Virtual Hosts

I have a manual setup on a dev server – so I needed to modify the /etc/php5/conf.d/newrelic.ini file to comment out the appname.

I could then add the variables into the /etc/apache2/sites-available/* apache config files with

php_value newrelic.appname "vhost.example.com;server.example.com"

WordPress W3 Total Cache

W3 Total Cache offers a basic integration with New Relic through its API.

There’s a couple of links for this:

W3 Total Cache assumes that you’ve setup the PHP application. What is non-obvious is that you need an API key and not your licence key. Via a WordPress SE answer, you can then enable API access (Select (account name) > Account settings > Integrations > Data sharing > API access) and use the generated API key in WordPress Admin > W3 Total Cache > General Settings > Monitoring section (wp-admin/admin.php?page=w3tc_general#monitoring).