# Functional Learnings of Python

Photo credit: cheers Hembo Pagi

This is my attempt at converting a solution to a pascal’s triangle problem that I’d written in an imperitive manner into a functional one. I kind of hope that anyone reading this is also trying to figure out the meanings behind functional programming, I’m trying to describe all the steps that I go through.

It is a mini way of me trying to discover what being ‘declarative’ actually means.

I know the kind of definition e.g.:

1. “say what you want” not “how to do it”
2. picture vs recipie
3. `sqrt(2)` vs looping from `x = 1` finding the mean of `x` and `2/x`
4. SQL, Haskell vs C++, Java

This is what I understand about functional programming:

1. no side-effects
2. focus on the functions (functions are first class objects)

## The imperative

Here was my imperative method to return the k-th row of Pascal’s triangle. Now I’ll be honest and say this took me hours to get a correct result when I first tried to implement it in an imperative manner.

```class Solution:
# @param A : integer
# @return a list of integers
def getRow(self, A):
pascal = [1]

for i in range(A):
pascal.append(0)
for j in range(i,0,-1):
pascal[j] += pascal[j-1]

return pascal```

## How do I say that declaratively?

It is supposed to be something like (see Functional Programming in Python):

```for thing in collection:
process(thing)```

Although I don’t really understand how that would be declarative. What’s my collection?

I kind of understand how SQL is declarative

```SELECT  field
FROM    table
WHERE   other_field > 5```

Instead of doing one big for loop.

What I don’t understand is how you can loop over an array looking at the current and previous elements.

To loop in a functional manner you need to be able to treat each element independently.

What I spotted was that you can copy and shift the array by one. So:

```#             current row = [1,3,3,1]
#      append zero to row = [1,3,3,1,0] (1)
#     prepend zero to row = [0,1,3,3,1] (2)
# sum elements of 1 and 2 = [1,4,6,4,1]```

This seems kind of neat, but I don’t really know how inefficient this is.

I also don’t know if that’s very functional. But I’m pretty sure that you can then loop over the elements of the arrays and sum them in a list comprehension, which always seems to be ‘functional’ (it’s final all encompassing functional solution suggested by this IBM article on Functional programming in Python).

The best I have so far is, for ‘what’ we want

```for i in range(k) # k-th row
current_row = get_next_row(current_row)```

But I couldn’t quite figure that out.

The next point that seems to be ‘ok’ with functional programming is recursion, which seems to be a sensible way of accessing current and next/previous

```def get_next_row(row, i):
if i = 0:
return row
return get_next_row(calc_new_row, i-1)```

At this point I turned to Stack Overflow. What I learned is that trying to search for functional programming solutions in Python isn’t very rewarding. Most people write imperitively in Python so you get a lot of chaff in your search results.

What made more sense was searching for Haskell solutions. The second solution I came across was Making a list of lists to compute Pascal’s triangle in Haskell, with this solution:

```nextRow vs = [1] ++ zipWith (+) vs (tail vs) ++ [1]
allPascals = iterate nextRow [1]```

Now you don’t have to know a ton of Haskell to understand the code. Also it helps if you know the `zip` python syntax which is similar, for summing the elements of two arrays.

`[x + y for x, y in zip(first, second)]`

Also this is following my concept of summing one array and itself shifted (by taking the ‘tail’):

```# [1,3,3,1]
# [3,3,1]
# [4,6,4]```

This gives you everything except the outer `[1]` elements, which get concatenated using the Haskell `++` operator (just `+` in Python). N.B. this also works because `zip` only processes until the end of the shortest list.

## The first attempt

So I combined this all into the following recursive function:

```def next_row(row, i):
if i == 0:
return row
new_row = [1] + [x + y for x, y in zip(row, row[1:])] + [1]
return next_row(new_row, i-1)```

The great thing was that this code was almost completely perfect at solving the problem the very first time I wrote it. The only problem I had was that I had mistyped the ‘tail’ and written `row[:1]` instead of `row[1:]`. Once I fixed that the code worked perfectly. Plus that was an easy, 5 minute bug to fix.

This is actually really, really exciting for me. To have code that works (passed the tests (correct results time and) and was accepted as a submission) first time off is awesome. The big time drain that I hit when solving coding problems was if your solution doesn’t work and having to bug fix things. This is what turns a 30 minute problem into an 4-10 hour problem.

## The improved attempt

N.B. Anyone learning from this there are definite issues with this section – I’m mutating `row`. See the last section for more details. I’m leaving this here though to show the ugly backward steps I made.

I still wanted to try and improve this though. I think a ‘smell’ in functional programming is if you end up with an if statement in your code.

So I made a futher attempt by introducing a `lambda`:

```class Solution:
def getRow(self, A):
process = lambda row: [1] + [x + y for x, y in zip(row, row[1:])] + [1]
row = [1]
for i in range(A):
row = process(row)
return row```

This code again, worked perfectly. It would have suffered from the same ‘tail’ bug I had previously, but prevents the possibility of bug in the base case of the recursion.

This then becomes more similar to the Haskell:

```nextRow vs = [1] ++ zipWith (+) vs (tail vs) ++ [1]
allPascals = iterate nextRow [1]```

## Conclusion

I still don’t really know if this is more ‘declarative’.

I’ve replaced

```for i in range(A):
pascal.append(0)
for j in range(i,0,-1):
pascal[j] += pascal[j-1]```

with

```process = lambda row: [1] + [x + y for x, y in zip(row, row[1:])] + [1]
for i in range(A):
row = process(row)```

It certainly is more similar to what the book suggested was ‘declarative’. But I could have just sub-functioned the inner for loop of the imperitive solution.

What it does do is eliminate the middle for loop and reduce the various `i`,`0`,`j-1`,`-1` elements which cause the bugs and are hard to track down.

In the functional code the only similar, ‘off-by-one’ style code is for `row[1:]` which is where my only bug appeared.

It is also useful that it effectively encourages you to create sub-functions.

So I still don’t know what declarative really is, but the code I seem to write searching for it seems more robust.

## Update

`row = process(row)` is not functional. It’s mutating row. I realised this, but as part of my learning I couldn’t think of a functional way of fixing it. Watching an OSCON video on Functional Thinking, I think that I actually need ‘reduce’ steadily build up the pascal row. This is work in progress…

# Jupyter on Windows with Chocolatey

## Install Python

This is a quick post (and will probably be outdated quickly), but it took me a while to get it correct.

I wanted to install Jupyter on Windows. The recommeded route is via Anaconda but I like to use Chocolatey the Windows package manager as much as possible.

I used Chocolatey to install Python package (currently v3.5.1). Assuming you’ve installed Chocolatey already, run as administrator:

``````choco install -y python
``````

This installs python to `C:\Program Files\Python35`. It also installs and `pip.exe` to `C:\Program Files\Python35\Scripts`.

Add `C:\Program Files\Python35` and `C:\Program Files\Python35\Scripts` to your `PATH` then you should be able to run `python` and `pip` immediately.

## Install PIP (hint: it’s already installed)

Note that pip gets automatically installed when you install python – nothing extra is required.

You can get led astray here by an official guide which suggests having to download and run `ez_setup.py`.

Don’t do this.

There is also confusingly a Chocolatey PIP package. Again don’t install this.

Otherwise you might end up here.

## Running PIP packages

Because we’re using Chocolatey, the installation doesn’t follow the recommended installation for Jupyter.

The main thing is that you need to add the python `Scripts` directory to your `PATH`, as above.

Install Jupyter

``````pip install jupyter
``````

This will install a `jupyter.exe` file into the `Scripts` directory. If that directory is in your `PATH`, then you should be able to directly run `jupyter` commands.

## Create a Jupyter notebook

Jupyter runs inside the browser but installs files and all the save data in the directory that you run the `jupyter` command from.

``````mkdir jupyter
cd jupyter
``````

If everything is working that will open up your browser at http://localhost:8888/.

# Segment Trees

Wow they seem overly complex to learn about. They’re a very easy concept, but there is a paucity of resources for explaining them.

Have a gander at the resources at the bottom for the ones I came across. Took me ages to figure out the differences between them.

Here, I assume you know about Binary Trees and recursion on Binary Trees.

Any code is in python which is about as close to pseudo code as you can get.

## tl;dr A segmented tree in one diagram

Structuring an array (length 8) in a segmented tree. The indexes of the array are the leaf nodes.

The parent nodes here are sums of the children (but could be another function).

This can be coded as a normal binary tree.

``````       28
/   \
6       22
/ \     / \
1   5   9   13
/ \ / \ / \ / \
0 1 2 3 4 5 6 7
``````

## Step 1: Create a tree with specific indexing strategy

We are going to use an example for storing `2^3 = 8` values indexed by [0,7] from an array in a segment tree.

We’re using 8 to ensure the segment tree we will create is a “perfect” binary tree to make things easier, or at least prettier. All segmented trees are ‘full‘ – all nodes have 0 or 2 children.

The values will get added as leaves of the tree.

So we want a tree that has 8 leaves.

To learn about how we construct them we’ll first number the nodes of a tree in depth wise (across the levels).

Some (Geeks for Geeks) start this at 0. League of Programmers start this at 1.

A perfect binary tree with 8 leaves has 4 levels and 2^4 – 1 = 15 nodes. The eight leaves are the values so we will use the 15 – 8 = 7 nodes for the segment nodes.

A binary tree with 7 nodes indexed by level, starting at 1 and with the 8 values of our array as the leaves:

``````       1 <= parent
/   \
2       3 <= i
/ \     / \
4   5   6   7 <= children
/ \ / \ / \ / \
0 1 2 3 4 5 6 7
``````

For each node `i` that we have indexed, e.g node `i = 3`, then

``````     parent = i/2 = 1 (integer division)
left child = 2*i = 6
right child = 2*i+1 = 7
``````

Alternatively, If you index the nodes from 0:

``````       0 <= parent
/   \
1       2 <= i
/ \     / \
3   4   5   6 <= children
/ \ / \ / \ / \
0 1 2 3 4 5 6 7
``````

Here for `i = 2`:

``````parent = (i-1)/2 (integer division)
left = 2*i + 1
right = 2*i + 2
``````

The difference between starting from 0 or 1 confused me a lot when I was looking at the various tutorials. I’m going to stick with indexing from 1.

## Step 2: Define what the segments are for each of the nodes.

`()` = ‘open’ boundary, `[]` = ‘closed’ boundary. We’re only using closed boundaries here.

``````x in (a,b] => a < x <= b
x in [a,b) => a <= x < b
N.B. [a,a] = a
``````

The root node `i = 1` refers to the whole array range [0,7]. Then node `i = 2` refers to the lower half [0,3] and node `i = 3` refers to the upper half [4,7]

N.B. [0,7] is the range 0-7 but it can also refer to a python list with 0 and 7 as values. Hopefully the context should make it obvious.

So, so far we have:

``````       1
[0,7]
/   \
2       3
[0,3]   [4,7]
``````

Then replacing all indexes of the tree with their corresponding segments:

``````         [0,7]
/     \
[0,3]       [4,7]
/   \       /   \
[0,1] [2,3]  [4,5] [6,7]
/ \   / \    / \   / \
0 1   2 3    4 5   6 7
``````

## Step 3: Create a function to store the segment of values

We want to set the value of the parent nodes to indicate the range of values that are in the leaves of the current sub-tree. This is quite well explained on Slide 16 of the League of Programmers’ slides:

“Each internal node represents some merging of the leaf nodes”

There are four common ones: sum, min, max, GCD (Greatest Common Divisor).

In more general terms, for node `i = 3`, `node value = f(a[4], a[5], a[6], a[7])` for the array `a`.

However this function `f` can be anything, e.g. other possibilities are count or finding the mid point of the range (`(max - min) / 2`)

Min is used for a the the Range Minimum Query, Geeks for Geeks have a Coded Example.

Example using sum:

``````       28 (= 6+22)
/   \
6       22 (= 9+13)
/ \     / \
1   5   9   13 (= 6+7)
/ \ / \ / \ / \
0 1 2 3 4 5 6 7
``````

This corresponds to the similar tree in the Geeks for Geeks tutorial.

## Step 4: Worked example to ‘fill in the blanks’

We’re going to use count as the function. We want to fill in the ith blank position (starting from 1).

We are given an array of [3,2,1,1] for which blank to fill for the following sorted values [1,2,3,4].

```    blanks = [3,2,1,1]
#         |
#         ^
values = [1,2,3,4]```

So 1 wants to go into the 3rd blank space. Then 2 wants to go into the 2nd blank space of the remaining 3 spaces.

``````Step 1: [_,_,1,_] <= 1 goes in 3rd blank space
Step 2: [_,2,1,_] <= 2 goes in 2nd blank space remaining
Step 3: [3,2,1,_] <= 3 goes in 1st blank space remaining
Step 4: [3,2,1,4] <= 4 goes in 1st blank space remaining

0 <= count of values in segment [1,4]
/   \
0       0
/ \     / \
1   2   3   4
_   _   _   _
``````

Fill in 1:

``````       1* <= count increased by 1
/   \
0       1*  <= count increased by 1
/ \     / \
1   2   3   4
_   _   1*  _ <= 1 placed in 3rd blank
``````

Fill in 2:

``````       2*
/   \
1*      1
/ \     / \
1   2   3   4
_   2*  1   _
``````

Fill in 3:

``````       3*
/   \
2*      1
/ \     / \
1   2   3   4
3*  2   1   _
``````

Fill in 4:

``````       4*
/   \
2      2*
/ \     / \
1   2   3   4
3   2   1   4*
``````

## Step 5: Code for ‘fill in the blanks’

We’re assuming that the tree has been created for us. The code for this is at the end.

To programmatically decide which blank to fill, we need to know if the ith value is in the left or right segment. Here’s the starting point again.

Initially, the counts are all zero. So for the root node we want to know if 3 is greater than or less than 4/2 (half the length of the array). If it’s equal, i.e. 2, then we want to put it in the left segment. Also if it’s less than i.e. 1 put it in the left half. If its greater than i.e. [3,4] then it wants to go in the right segment.

```    N = len(segment)
if N/2 < i:
# go right
else: # i <= N/2
# go left```
``````       0 <= looking for 3rd leaf node, i = 3 > N/2 so look right
/   \
0       0
/ \     / \
1   2   3   4
_   _   _   _
``````

At the second level, the count is still zero. But now we are only considering half the values. so `N = 2`, but now we want to know if the initial `i = 3` goes in the 1st or 2nd segment. We’ve discarded half the values so we can subtract half the original `N = 4` values from i. So at the first level of recursion, we alter `i` for the next level to be, `i = i - N/2`. Going left requires no change.

The length of the segments, if even, will divide in two, so `N = N/2`. If odd, then `N/2` (using integer division) go left and the rest (`N - N/2`) go right so we split range [1,N] into [1,N/2] and [N/2+1,N]. So [1,3] => 1 and [2,3]

``````       0
/   \
0       0 <= i = 1, N = 2, i <= N/2 so look left
/ \     / \
1   2   3   4
_   _   _   _
``````
```    def fillBlank(self, segment, i, N):
# N = len(segment) # we pass this in
if i <= N/2:
# go left
N = N/2
self.fillBlank(segment.left, i, N)
else: # N/2 < i
# go right
i = i - N/2
N = N - N/2
self.fillBlank(segment.right, i, N)```

Then when we hit the leaf then we can set the value.

```    # N.B. the new params blanks and value
# blanks is passed in by reference
# value is going to fill the blank
def fillBlank(self, segment, i, N, blanks, value):
if not segment.left and not segment.right:
# N == 1
# we're dealing with array indexes, so minus 1
blanks[segment.value - 1] = value
return
# ...```
``````       0
/   \
0       0 <= i = 1, N = 2, i <= N/2 so look left
/ \     / \
1   2   3   4
_   _   1   _ <= blanks[2] = 1
``````

Now, to complete our first pass through the tree, we should have updated the counts (the values of each of the nodes) by one as we passed through it. So the full method becomes:

```    def fillBlank(self, segment, i, N, blanks, value):
if not segment.left and not segment.right:
# N == 1
# we're dealing with array indexes, so minus 1
blanks[segment.value - 1] = value
return
# increment the count of this segment
segment.value += 1
# N = len(segment) # we pass this in
if i <= N/2:
# go left
N = N/2
self.fillBlank(segment.left, i, N, blanks, value)
else: # N/2 < i
# go right
i = i - N/2
N = N - N/2
self.fillBlank(segment.right, i, N, blanks, value)```

However this doesn’t quite work for all the following passes through the tree, e.g.searching for the 1st blank space for the very final step – which wants to go into the blank space which is at the end of the array.

To do this we take into account the count of the child segments. If the left segment has been filled up but the right segment is empty, then we automatically want to look in the right segment.

So the condition `i <= N/2` becomes `i <= N/2 - X` where `X = segment.left.value` and when going right, we alter the ith element similarly `i = i - (N/2 - X)`.

Finally we don’t care about this conditional check when the segment only has two values in it as those two values are just the leaves (which don’t contain counts). This works since all segmented trees are ‘full’ (every node has 0 or 2 children).

```    def fillBlank(self, segment, i, N, blanks, value):
if not segment.left and not segment.right:
# N == 1
# we're dealing with array indexes, so minus 1
blanks[segment.value - 1] = value
return
# increment the count of this segment
segment.value += 1
# N = len(segment) # we pass this in
# this section gets more complicated from edge cases
if N > 3:
X = segment.left.val
elif N == 3:
# left node is leaf
X = segment.val - segment.right.val
elif N == 2 and blanks[segment.left.val - 1]:
# need to check if left has been filled
X = 1
else:
X = 0

if i <= N/2 - X: # X subtracted
# go left
N = N/2
self.fillBlank(segment.left, i, N, blanks, value)
else: # N/2 < i
# go right
i = i - (N/2 - X)
N = N - N/2
self.fillBlank(segment.right, i, N, blanks, value)```

Here’s the first step of the second pass:

``````       1 <= looking for 2nd leaf node, X=0, N=4, i=2 <= N/2 so look left
/   \
0       1
/ \     / \
1   2   3   4
_   _   1   _
``````

We then follow the recursion through, which should populate the `blanks` list.

# Lock Stock Pomodoros

I give you http://pom.ianchanning.com. Pomodoros with a Lock Stock film quote at the end of each one.

I’m a little obsessive about timers (see bottom) and the other pomodoro timers I’ve tried failed on different counts.

I’ve tried to cram everything I find useful about a timer.

Firstly spacebar to start/stop. The timer should be something that sits in the background and requires minimal attention. So a quick ctrl+tab to get the the pomodoro tab and then hit space and then back to your work.

Secondly the timer appears in the title bar and it appears first. Then you can easily see that it’s running so that you can still see the timer when you have lots of tabs open.

Thirdly there’s a start and stop. Possibly this flies against what a Pomodoro timer should do. However any timer I have I want to be able to start/stop/reset. That’s just what a timer should do, it’s annoying if it doesn’t.

Fourthly the timer has to notify you properly when the pomodoro finishes. This is achieved through a buzzer and a browser notification. Notifications sometimes don’t work or you don’t see them so you need the buzzer backup. Sorry IE folks your browser is still terrible and can’t handle the Javascript Notifications and I can’t be bothered to fix it when it’s just for a personal project.

Fifthly a count of the pomodoros is useful to know how many you cranked out in the day.

Sixthly, a bit of browser responsiveness goes a long way.

And that’s it. No more please.

Well except I needed to put something in the notification so I jazzed things up with quotes from Lock Stock and Two Smoking Barrels. If you can’t take the ‘c’ word at least once then this timer is not for you.

For those who got this far. I started with the most basic timer I could think of that still had a simple start-stop mechanism. Then after staring at the jquery code I’d copied I realised I could make a generic jquery ticker function. So I can tick 25 minutes down and then tick the pomodoro count up 1 each time. The rest is a codged together front end on top of that.

# Linux Mint Dual Boot

It’s not supposed to be this painful.

I finally (9 months after first attempt) got a UEFI dual boot working.

I’m calling Linux Mint my lucky charm as that was the first one I got working as a Vagrant desktop and it’s worked perfectly now that I’ve sorted out the other issues.

Having read and re-read so many articles – there were three final things I had problems with/didn’t understand:

1. Unetbootin was creating blank USB drives fixed by using universal-usb-installer as recommended by both articles above
2. UEFI bootable USB stick Universal USB works fine (you can create a UEFI only one by simply extracting an ISO using 7-zip and just copying the files, which ignores the extra step that also installs a MBR file (I think))
3. Booting to USB when the BIOS is set to UEFI In Windows hold shift and click ‘Restart’ (fucking magic) – this point is buried as almost an aside in the main itsfoss.com article above:

To boot from USB, will have to choose boot from USB option from within Windows itself. Either with PC Setting (like for UEFI) or pressing shift key while clicking on Restart.

Point 3 is also in the wonderfully in-depth tweaking4all.com article.

The rest seemed quite ok as I had the rest of the BIOS settings / fast boot options correct.

Once I got the USB stick working, I ran the check disk once and then went into the live Linux Mint. The first happy thing was that the Wifi was working straight off – which wasn’t before with Ubuntu. I then followed the tweaking4all guide to create:

• first a root (/) /ext4 logical partition
• leaving 8GB (8191MB, to equal the RAM size) of space for the swap logical partition.

After Linux Mint installed – basically everything was perfect. Where as the tweaking4all guide talks about running the ‘Boot Repair’ program from Mint before restarting. However I took a gamble based on the itsfoss article that a reboot should work. And it did.

The GRUB bootloader was happily installed into the EFI ‘partition’ (not sure if that’s the right word) so that now starting the computer shows the dual-boot screen.

Then booting Linux Mint works perfectly fine – and again the Wifi was working. As was the other major obvious one of the power button working.

As a minor anecdotal point – the booting times (Lenovo Yoga 13, Core i7, 8GB RAM, SSD) for Mint vs Windows was 20s for both to get to the desktop after logging in. There goes my theory of how gloriously fast Linux is to load. I guess that just isn’t the case with Ubuntu based systems.

Now to blat Mint and see if Fedora is any faster.

# Scrum unsucked

[Photo credit: darkmatter]

I’ve been reading into Scrum closer recently and it just seems like a horrible corruption of an inspirational idea, that could much more easily be integrated into normal project practices.

## Agile

I love the agile principles. I’m repeating them here for my own benefit, I like a good list as much as the next.

1. Customer satisfaction by rapid delivery of useful software
2. Welcome changing requirements, even late in development
3. Working software is delivered frequently (weeks rather than months)
4. Working software is the principal measure of progress
5. Sustainable development, able to maintain a constant pace
6. Close, daily cooperation between business people and developers
7. Face-to-face conversation is the best form of communication (co-location)
8. Projects are built around motivated individuals, who should be trusted
9. Continuous attention to technical excellence and good design
10. Simplicity—the art of maximizing the amount of work not done—is essential
11. Self-organizing teams
12. Regular adaptation to changing circumstances

Each principle makes sense, even if, perhaps, it isn’t attainable.

Their manifesto is harder to figure out, but still has a great aspirational feel to it.

We value this Over this
Individuals and interactions Processes and tools
Working software Comprehensive documentation
Customer collaboration Contract negotiation
Responding to change Following a plan

However importantly note that the second column is still valued, but just not as much as the first.

## Scrum

Now read the Scrum Guide™ (TM!). Scrum just seems like a conversion of agile concepts into sucky word project management.

I realised I’d already got it wrong with what a sprint was – I thought it was two weeks work. They suggest that it should be a month’s worth of work – which makes sense, but why not just call it a month’s worth of work – or if it’s going to be less just call it an increment.

Anyway I read through the Scrum Guide and here’s my unsucked version of their main concepts.

Scrum Guide™ Unsucked agile guidelines™
The Scrum Team The Team
The Product Owner The Client
The Development Team The Developers
The Scrum Master The Project Manager
The Sprint The Sprint (it isn’t actually that bad a word) / Increment
Sprint Planning Planning
Daily Scrum Talk
Sprint Review Sprint Update
Sprint Retrospective Sprint Review
Scrum Artifacts Documents
Product Backlog Issues
Sprint Backlog Sprint Issues
Increment Completed Issues
Artifact Transparency Simplification
Definition of “Done” Happy Client

## Can I have a word…

### Scrum style

“Shall we go over the Sprint Review later? I’m not convinced that the Product Backlog contains all that we require for the definition of done. The Daily Scrum this morning wasn’t great as the Scrum Master had to get the Development Team to better explain their progress to The Product Owner. We need to make the Sprint Backlog artifact more transparent as The Product Owner feels we haven’t followed what we agreed at the last Sprint Retrospective.”

### vs sane

“Shall we go over the Sprint update later? I’m not convinced that the issues contain all that we require for the client to be happy. The talk this morning wasn’t great as the Project Manager had to get the developers to better explain their progress to the business. We need to simplify the Sprint’s issues as the client feels we haven’t followed what we agreed at the last Sprint review.”

Please let the first type of conversation cease.

# New Relic monitoring for Server, Virtual Host PHP and WordPress

New Relic offers completely free server monitoring (CPU, RAM, Network I/O etc) and limited (24 hour retention) PHP application performance monitoring (detailed error logs, PHP vs MySQL load times etc). Its far better than monitoring tools I’ve seen and very useful for trying to figure out any problems with WordPress which can be quite heavy in different areas CPU vs RAM vs Database.

## Server monitoring

New Relic have excellent docs on the server installation, but this is a summary of the commands, N.B. the `YOUR_LICENCE_KEY` needs to be changed:

```sudo sh -c 'echo deb http://apt.newrelic.com/debian/ newrelic non-free > /etc/apt/sources.list.d/newrelic.list'
sudo apt-get update
sudo apt-get install newrelic-sysmond
/etc/init.d/newrelic-sysmond start
```

## PHP monitoring

There’s again useful New Relic installation notes:

```sudo sh -c 'echo deb http://apt.newrelic.com/debian/ newrelic non-free > /etc/apt/sources.list.d/newrelic.list'
sudo apt-get update
```

This is all repeated from the Server section, so the only actual line required is:

```sudo apt-get install newrelic-php5
```

This asks you to give an Application Name – the best format to use is `[vhost domain];[server domain]` where the server domain is used as a kind of parent filter for all vhost domains.

### PHP vhosts applications

This doesn’t quite work for Plesk installations as we want a per-directory application performance indicator. This assumes that we’re using the Apache php mod rather than FastCGI as we’re putting PHP ini settings in apache config files.

#### Plesk

This guide (http://blogs.reliablepenguin.com/2015/02/11/plesk-new-relic) was the perfect starting point. The application names they suggest are really useful e.g.

```[vhost domain];[server domain]
```

#### Changing the application name

If you make a mistake with the application name you can change it. Just follow this document (https://docs.newrelic.com/docs/apm/new-relic-apm/maintenance/renaming-applications).

#### Apache Virtual Hosts

I have a manual setup on a dev server – so I needed to modify the `/etc/php5/conf.d/newrelic.ini` file to comment out the appname.

I could then add the variables into the `/etc/apache2/sites-available/*` apache config files with

```php_value newrelic.appname "vhost.example.com;server.example.com"
```

## WordPress W3 Total Cache

W3 Total Cache offers a basic integration with New Relic through its API.

There’s a couple of links for this:

W3 Total Cache assumes that you’ve setup the PHP application. What is non-obvious is that you need an API key and not your licence key. Via a WordPress SE answer, you can then enable API access (Select (account name) > Account settings > Integrations > Data sharing > API access) and use the generated API key in WordPress Admin > W3 Total Cache > General Settings > Monitoring section (wp-admin/admin.php?page=w3tc_general#monitoring).

# Second hand bicycles

In response to a blog post ‘Why I keep fixing my bike‘ about the risks of buying a dodgy second hand bike, I wrote the following comment. It’s something I’m quite passionate about so worth storing on my own blog…

I have bought at least 7 bikes second hand through either ebay or gumtree all without problem. Every bike has been exactly what I was looking for. I wouldn’t buy off an ebay seller with only a small repuatation – and almost all sellers I have seen when searching ebay have 100+ reputation.

I think the chances of getting a lemon are quite small, but you always have to be careful.

With a bicycle you are predominantly paying for a decent frame as the rest can be replaced. Its fairly unlikely that the frame will have been damaged. So buy bikes that have well made frames e.g. Cannondale. I’ve purchased a Specialized, Cannondale, Merkx, Orange, Klein and Marin bikes. Most of the bikes I bought would have had a new value in excess of \$1000. I’d be most wary of buying a Specialized bike as they are the most common and the current Specialized bikes I’ve seen have very poor welding on the frames.

Certainly don’t go buying cheap branded bikes new bikes e.g. and shop’s own brand. If you’ve got very little money you can pick up old retro racing bikes for less than \$100.

Limit your search to only bikes that are near buy (i.e. 30 minute drive) so that you don’t waste much time.

There are many things you can look for:

1. Has the bike been stored inside – if you see a photo of it hanging up in a garage on its own rack its a pretty good bet that it has
2. Buy ‘fixed’ bikes without suspension as the suspension can be expensive to replace
3. Look at the type of gears that it has
4. Check how detailed the spec of the bike is – does the person list exactly all the components – this gives an indication of how much the person cares about the bike
5. Do they state where they bought it from
6. Do they display high quality images of close ups of the gears, the joins of the frame where cracks appear
7. What have they been using the bike for
8. How often do they clean the bike
9. Ask what year the bike is from as well as the exact make and model – then hopefully you can find a reference to how much the bike was when it was new.
10. Check the welding on the bike frame – how clean are the welds. You shouldn’t be able to see the welds – as that means it was a cheaply made bike by machines and never finished off.

# Dos command history

One of the nice things of Linux is the command history – it stores typically the last 1000 commands or so and its saved every time you log out.

DOS does have this through the `doskey /history` command (but only for current session) and you can relatively easily append this to a file:

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.

 doskey /history >> %USERPROFILE%\history.txt%

view raw

doskey.bat

hosted with ❤ by GitHub

The tricky bit is having this done for you on log out.

Via a Stack Overflow question on bash_history you can get close, using doskey macros which appends to a file on `exit`.

This only works for the current session though so you need to create a shortcut that runs the command each time with the following Target:

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.

 %windir%\system32\cmd.exe /K doskey exit=doskey/history\$g\$g%USERPROFILE%\history.log\$texit \$1 \$2

Then you just have to remember to type `exit` each time…

### Update

I also found three useful commands – but they’re only for your current session. Hat tip to Nifty Computer Tricks:

F3
get last command
F7
current list (use left/right arrow keys to paste to the command line)
F8
auto-complete (type the first few letters)

### Update 2 – Some extra love for the DOS prompt…

If you’ve got this working then I’ve added some extra basics. First you need to install Git (and I suggest you do it using the Chocolatey package). This includes the best versions of gnu commands compiled natively for Windows.

Add the Git usr bin directory to the path e.g. `C:\Program Files\Git\usr\bin`.

Put the `!.bat` and `history.bat` into the Git usr bin directory too. These assume you are saving the history to `%USERPROFILE%\history.log`.

Then you’ll be able to do commands like `history | grep mysql` to list all your mysql commands in `history.log` with their line numbers. Then call `! 123` to execute the specific line of the `history.log` file.

### Update 3 – Clink love

Martin Ridgers’ Clink is an awesome tool to behold. That adds the correct `!` command to the DOS command prompt. Buuuuuuut… it doesn’t include a `history` command. So save the above `history.bat` file to `C:\Windows\System32\history.bat` (it doesn’t have to rely on GNU Tools / Git) and uncomment `history_file=%localappdata%\clink\.history`.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.

 @echo off rem simple replacement for linux ! syntax e.g. !591 rem usage: ! N (executes line N of history.log) if "%1"=="" goto Syntax if "%1"=="/?" goto Syntax for /f "tokens=*" %%x in ('sed -n "%1{p;q}" %USERPROFILE%\history.log') do echo %%x & cmd /c %%x goto End :Syntax echo usage: ! N :End

view raw

!.bat

hosted with ❤ by GitHub

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.

 @echo off rem replacement for linux `history` command rem usage: history set history_file=%USERPROFILE%\history.log rem For if you are using Martin Ridgers 'clink' rem @link http://mridgers.github.io/clink/ rem set history_file=%localappdata%\clink\.history rem this doesn't require GNU Tools, but doesn't have line numbers rem type history_file rem requires GNU Tools installation (e.g. Git SCM) cat -n %history_file%

view raw

history.bat

hosted with ❤ by GitHub

# OpenVPN connects to VPN but no internet

For those stumbling down a similar path…

tldr; Check that your Windows Firewall is turned off.

## My setup

• OS: Windows 8.1 64-bit laptop
• OpenVPN: `OpenVPN 2.3.4 x86_64-w64-mingw32`

My OpenVPN has been working for at least 4 months without problem.

First thing to remind yourself is: When you get the VPN working – copy the OpenVPN log of a successful connection and the `ipconfig /all` to file so that you know what to compare when it breaks next time.

## Notes

• The VPN connection stopped working yesterday 17/10/2014
• Not exactly sure when
• It had been working at my office
• My Windows 7 desktop has a working VPN setup so I can compare the settings
• The OpenVPN gui suggests a connection has been made but no sites can be pinged (see commands below)
• All attempts at `inconfig /renew`, `ipconfig /flushdns` nothing worked
• Only differences between Desktop/Laptop I could find in `ipconfig /all` and OpenVPN logs was that laptop had IPv6 for the WiFi connection turned off and the following extra fail message `UDPv4: No Route to Host` in the log
• I tried turning IPv6 back on but that didn’t change anything
• Now the UDPv4 message disappeared – but didn’t reappear when I turned off IPv6 again – so that was a red herring
• Possible suggestion from my history when first setting up my VPN (http://www.rickygao.com/troubleshooting-general-failure-in-ping-or-tracert/) was to turn off IPv6 which is what I’d probably done before – but IPv6 is no problem on my Windows 7 desktop
• Pinging even a specific IP address does not work – this confirms its not a DNS issue
• Pinging localhost `127.0.0.1` does work
• Pinging the IP Address of the connected VPN does work
```C:\Users\Ian>tracert google.com
Unable to resolve target system name google.com.

Ping request could not find host google.com. Please check the name and try again.
```

## Solution

Eventually found a solution that was to turn off the windows firewall, which came from this unbuntu forums post which eventually spotted that it was an `iptables` firewall issue.

## Post mortem

• I thought I’d turned off my firewall and it doesn’t explain why the VPN stopped working
• Best guess is a Windows update changed something in the firewall the caused the problem
• A failed update for `Windows Malicious Software Removal Tool for Windows 8, 8.1 and Windows Server 2012, 2012 R2 x64 Edition - October 2014 (KB890830)` – this was successfully installed the next day but didn’t help
• A successful `Definition Update for Windows Defender - KB2267602 (Definition 1.185.3528.0)`