Category: Projects

What Makes an Application Scalable?

December 6th, 2021
What Does a Scalable Application Mean?

Applications that are depended on by thousands of users may see peaks or dips in demand during the day and managing the cost of running the infrastructure can be challenging. Scalable applications are applications that are able to increase their resources to serve requests as they come.

What types of Scaling are there?

There are two basic examples of scaling for applications,
- Vertical Scaling, where an application’s resources are increased due to demand, such as increasing the RAM available to an application host. Vertical Scaling is also sometimes referred to as scale-up.
- Horizontal Scaling, where an application is spread across different nodes in a cluster. This is most appropriate for applications that require a lot of resources. Horizontal Scaling is also referred to as scale-out.
Scalable applications have many uses, including;
- Allowing cost reduction during low utilisation by scaling down clusters or pods serving the application.
- Improving quality of service during peak load times by scaling up horizontally or vertically by utilising resources made available to the application autoscaler.This will ensure that your application always has the right amount of capacity to handle the current traffic demand.
- Faster processing as features can use the optimal storage technology for data and information.
Best Practice for Scalable Applications

Applications usually are best compartmentalised into components for both design and maintainability, monolithic architectures for code bases and applications have caused application elements to become obscure and entrenched, using a more distributed approach from the start can reduce cost and technical debt as components can be re-written or swapped out.
- Splitting transient application functions in to their own libraries or APIs can allow the front end and back end to operate independently, this can allow processes that take time to be acted on based on events rather than cause waiting or processing as a batch.
- Data storage should be independent to the service that owns the data and therefore should use the best storage type and mechanisms available such as caching or streams of data. Slow returns should be accounted for and independent from the user interface.
- Applications should not be strained by resources, whenever possible applications should perform the same for every user or function regardless of workload or work-type. Rather than wait for functions to complete, have a common goal of eventual data integrity.
- When implementing concepts like micro-services you should endeavour to ensure standard practices like a common set of languages or behaviours to improve maintainability.
- Complexity can sometimes be harder to manage than an equivalent monolithic application, even though each function should be simpler.
- Isolate failure to individual services.
Recovering my Data after Drive Failure vs AWS S3 Glacier Deep Archive

November 16th, 2021
I recovered my data from AWS S3 and all I got was this lousy bill.
Aidan – Alternate Headline.

TLDR;
- One of my hard drives failed, I thought I’d try to recover the valuable 400GB using ddrescue, it sort of worked.
- Restoring from S3 is expensive £27.53 for ~400GB
The Scenario

A week or so ago I realised that my hard drive was on the way out, its been on for almost 27,000 hours according to the SMART data. I first noticed when the PC was loading into check disk after every reboot. It took me about 3 reboots to decide something was up and I used Crystal Disk Mark to check the disk and sure enough it was reporting ‘Bad’. So I ordered 2*6TB drives and thought I’d better have a go at moving the data and making sure my backups were up to date.

For my backups, I use cloudberry backup (now called something else) which is an encryptable cloud backup solution which is compatible with Amazon’s S3. I use the cheapest storage option, S3 Glacier Deep Archive.

DDRESCUE GUI Data Recovery

DDRESCUEVIEW Preview

I booted in to a persistent live Ubuntu 20 environment and installed ddrescue, ddrescueview and ddrescue-gui. I found that the tools worked well but took way to long for the drive, you can see in the remaining time section of ddrescue-gui it would have taken an estimated 60 days to recover the data at the fastest setting.

Making DDRESCUE Faster

To make ddrescue faster I found it was best to watch the drive speed in ddrescue-gui and then I scrapped it over the command line for a faster experience.

In the end I used these commands, make sure to replace the drives with your setup and the minimum read rate to one your drive is comfortable with. For the first command, I stopped it at around 90 percent of the way through the drive and swapped it for the second one.
```
# First run to cover myself in case the drive died more seriously.
sudo ddrescue -f --reopen-on-error --min-read-rate=8000 /dev/sdd2 /dev/sdc1 /home/ubuntu/Documents/log1.log

# Lots of Passes to try to recover slow sections.
sudo ddrescue -f --reopen-on-error --retry-passes=5 /dev/sdd2 /dev/sdc1 /home/ubuntu/Documents/log1.log
```
Although this really seems like a your mileage may vary environment depending on the type of failure your drive has.

If you do end up using ddrescue-gui at least to begin with, you can use the log file to get you a command to start off with. Make sure to read the manual pages for ddrescue to determine the best command for you.

Here is an example of one of my outputs (.log files),
```
# Mapfile. Created by GNU ddrescue version 1.23
# Command line: ddrescue -v -d -f -r 4 -c 128 /dev/sdd2 /dev/sdc1 /home/ubuntu/Documents/log1.log
# Start time:   2021-09-30 21:55:30
# Current time: 2021-09-30 22:05:21
# Copying non-tried blocks... Pass 2 (backwards)
# current_pos  current_status  current_pass
0x1C63D4C0000     ?               2
#      pos        size  status
0x00000000  0x1E160000  +
0x1E160000  0x00010000  *
0x1E170000  0x01320000  ?
0x1F490000  0x03540000  +
0x229D0000  0x00010000  *
0x229E0000  0x01320000  ?
```
You can of course view this data using ddrescueview.

ddrescueview

DDRESCUE Results

After a week and a bit, I decided to stop the experiment and see what had been recovered. ddrescueview looked like this,

ddrescueview results

ddrescue was able to recover about 90.83% of the ntfs partition, enough to mount the drive and view the data. It contained many of my important personal files and more importantly images and home video. The actual used space on the drive was only ~700GB, with around ~450GB of data that was valuable to me.

When I opened the personal photos and videos, I found the results to be quite poor, there were glitches in them, sometimes files had no actual data in them, sometimes they had stripes and lines in the image, because of the spread of the failure across the partition blocks, the data was basically a really poor copy with a lot of holes.

ddrescue non-tried sectors made many files unreadable or poor quality

I decided that it was best not to continue the recovery with ddrescue and instead restore from backup, the age of the backup was exactly 1 month prior to the failure to the day, so no real loss. However only the data that I truly cared about was backed up. So stuff like my VMware ISO files and downloads folder were lost and unrecoverable.

Downloading from AWS S3 Glacier Deep Archive

Using Cloudberry I made a restore plan and recovered the data using the slowest SLA at 3-5 days, which by sods law took the full amount of time to process and then some, because I put in the wrong decryption password and needed to re-request the data.

Anyway here is the bill, £27.53

AWS S3 $37.31 (£27.53) Bill

The killer was the data transfer fees out of London, at a cost of $0.09/GB ($28.37).

And with that, all of my data was re-recovered, this time without corruption.

Learnings and Final Thoughts
- Although AWS S3 is a valid backup option, its expensive to recover from. I already pay roughly $1/mo for ~400GB (315GB compressed). For larger recovery this would be prohibitively expensive, multi-terabyte or whole disk backups would require compression.
- Physical damage to a hard drive is essentially game over, your data is lost. For best results have redundancy. This is the only reason I am thankful for S3, it was my only solution to recover my data. A local backup would have been much cheaper and faster to recover.
- The two new 6TB drives run in a Windows storage spaces two-way mirror pool.
Windows Storage Spaces
Flashcard Club V0.0.3

May 10th, 2021
Its been a good while now and Flashcard Club is well on its way to a functional product. There has been some progress on features, notably the inclusion of Google OAuth2 through Laravel Socialite.

Logging in through your Google Account should greatly increase the speed of adoption for new users and improve retention and user acquisition.

With that in mind, Flashcard Club has been online now for about 3 months and has yet to have a single user. I believe this is mostly due to myself not promoting the product which I will feel more comfortable doing once the site is ready, however it is currently somewhat usable. To that I have made improvements in the homepage, mostly bringing it up to paces with a call to action and have moved the changelog to an FAQ page, which I may change later as I do not love the name.

Flashcard Club’s V0.0.3 Homepage

Flashcard Club’s Login Options

Google OAuth2 Registration option

Version 0.0.3
- User Accounts
  - Users can now login and link their Google Account to Flashcard Club.
- Landing Page
  - The Landing page has had a massive makeover and most of the content is different now.
  - The changelog has been moved to the FAQ page.
- Test and Study Mode
  - There is now a chart to plot test performance per set.
  - Test mode now has additional functionality like gold highlight on completion of the test.
  - Test Mode now has a summary.
- Planned Features
  - ~~Google Sign in (Federated identity)~~ Completed
  - Terms of service
  - Privacy Policy
  - ~~FAQ Page~~ Somewhat Completed
  - Markdown User Guide
  - Flashcard User Guide
  - ~~Front page needs work~~ Not complete but looks a lot better.
  - Cramming mode that removes cards previously marked “Correct”.
Part of Improving the site in the next round will also be improvements to the mobile aspects of the site as most users will likely be on mobile devices.

I have also been ignoring the fact there is currently no export option available to users.
Flashcard Club February

February 4th, 2021
It’s been a little while since the 25th of December when I decided to go public with this idea, a great deal of work has gone into flashcard club. Exactly 40 hours has now been dedicated to the project outside my normal 8:30-5.

It’s been quite slow going in some aspects, one particular problem is the markdown library options are numerus and vary in completion and schema that make it hard to choose one that is appropriate for this project. In the end I have settled on marked.

Another area I am struggling to decide on is how best to onboard new users, I want them to have a positive experience however not all know the markdown syntax.

Currently, Flashcard Club has the following features,
- User Accounts,
  - It’s currently possible to create a new user account, although there is no ability to ammend or update your user account.
  - New users should recieve a welcome email.
  - New users should be logged in automatically.
  - New users should be able to create flashcards and flashcard sets.
- Almost Unlimited Flashcards,
  - It is possible to create and save flashcards.
  - You can also update flashcards, delete them and more.
  - Flascards use markdown syntax so you can include pictures and a limited amount of html.
- Unit Tests,
  - Many components of Flashcard Club have PHPunit Unit tests.
  - Code coverage is probably very poor.
- Toast Notifications,
  - Toast notifications are intergrated into the website.
  - The best place to see these in action is in the set editor if you update the title.
Planned features include,
- A Study mode, because currently it is only possible to edit flashcards but not view them without the editor.
- Google Sign in (Federated identity).
- Statistics, because who doesn’t love a good graph.
- Account management, because currently once you make an account you cant change any of your details, oops.
Currently statistics and study mode do not work.

Thanks for reading,

Aidan.
I’m starting flashcard.club

December 24th, 2020
Hello everyone,

I’m sure there is not many of you but I had to tell someone. As of the last few days I’ve really felt the need to get involved in something that is really what I am all about. A new baby if you will, a commercial enterprise but one I own. I’ve been working full-time on Laravel projects and I want to improve my skills to really get better at it. Its a great framework after using raw PHP and template engines like handlebars.

During this time I came to the realisation that I wasn’t totally stoked with the existing flashcard companies out there and I think now more than ever I really have the skills to get something like this off the ground, hopefully this is the first project of many!

Flashcard Club is different,
- I really want a modern material feel to the website. I’m not interested in something worse than what there already is out there, I want to really help people.
- I want to really build a solid brand that people want to use. Flashcard Club should be a place to really get some work done and cram for that final test and study in a safe space.
- I really want to provide a platform for students to get some work done. A lot of the competition has low quality submissions. I really want people to have an effective space to learn.
- I want to publicise what I’m up to, and this blog will help with that. A real leading learner approach. From conception to reality.
- Although this will have options to buy products, long term I only want this project as a sort of demo of my capabilities. I want to move on to better ideas ASAP but you have to start somewhere.
This is my first “premium domain” project so consider that an invested interest in this project.

Have a look and see how I’m getting on. At the time of writing cloud flare gives out error 526 so I hope to be past that by the time this goes up a month from now.

Flashcard Club

I aim to make both a website and mobile app, however the majority of the features will be made through the app, the functionality and backend will be possible through the online version as well.

I think my target audience is 99% students so a strong app presence is a must I feel. The website will be a strong start but is just one part of the experience.

Watch this space.

Aidan.
Hull University Minecraft Map

August 4th, 2020

I used geocraft to generate a Minecraft map of the University of Hull, and now I am sharing it with you. You can see a short video of the map below.

The map is available as OpenStreetMap data only, or with the DEFRA (Department for Environment, Food & Rural Affairs) lidar data from, (I think) 2015, so the multi-story car-park is not included for example.

The original project is broken as the lidar data did not get included, but it was possible to fix it using the suggested changes by RobinWhitfield here in this issue.

The original licencing for this map is CC-BY-SA for the pure OpenStreetMap map and the other is the open government licence. You may use these maps as you wish within these constraints.

If you plan on running these maps on a server, you may want to turn off leaf decay as some of the trees are all leaves and do not have a wood block to prevent them from decaying, as seen in the video.

HullUniversity LIDAR Full Map with Buildings

HullUniversity OSM Full Map without Buildings

I am not associated with Hull University, they’re good guys.
Check all API variables are included in a REST API for PHP.

July 1st, 2020
I recently have been working on a weather recording project and as part of this endeavour wanted to check that all of the variables I have posted to my API were set, it can be rather a lot of work to have to specify each if (!isset($_GET[])) parameter so I decided to use a function that can take any number of arguments using a ‘variadic’ function in PHP.
```
// Returns true if all variables are set, else returns false.
function getVariablesSet(string ...$getlinks) {
      foreach ( $getlinks as $link ) {
        if (!isset($_GET[$link]))
        {
            return false;
        }
    }
    return true;
}
```
Now, when I want to use a new api ‘command’, I can simply do the following,
```
$browser_response = new stdClass();
$browser_response->message = "Command not specified.";

if ($_GET['command'] == "new-temperature")
    {
        if (getVariablesSet("datetime","temperature","humidity"))
        {
            $browser_response->message = "All GET variables set.";
        }
        else
        {
            $browser_response->message = "All GET variables not set.";
        }
    }

echo json_encode($browser_response);
```
This way, so long as we have specified the command and the variables required we can enter the scope, otherwise, we can kick them out until their query is formatted correctly. Having lots of variables may become problematic so you may want to use POST or even break them out into subsections to give users a better understanding of their error. I should add this only works for PHP 5.6 and above.

In any capacity, Good Luck. Aidan.

My Home Network – Part 2

June 1st, 2020

Hello again, this is the second in a series of my home network posts. In this edition, we’re going to be flashing another router with OpenWrt. You can read part one here.

I decided it was time to leave all the old hardware behind and move to brand new stuff. I was having problems with the ZYXEL VMG8924-B10A as the main router, it kept cutting out and was causing short minute outages that ultimately I think was due to the system running out of ram and botnets trying to break into the thing. So I decided it was time to jump ship and move to something a little bit more enterprise. At the same time, I decided now would also be a good time to leave the Netgear WNR3500L V2 to one side. Despite it serving us good for many years, its routing features won’t be necessary for the upgrade as I want to have a good crack at managed switching.

The new network

The new network is composed with the following,

A Ubiquiti EdgeRouter X
A Netgear ProSafe GS108T
A new to me MR33

I wanted to do away with having two routers on the network, it wasn’t neccisary and caused some of the upstairs equipment to be inaccessible from downstairs, the new configuration would mean that all management would be done through the EdgeRouter X which would be much cleaner and hopefully faster.

The ProSafe GS108T was a garage find, I wasn’t using it for anything and I hadn’t really implemented proper VLAN tagging before so I thought now was a better time than ever to get into it.

Network Topology — My new home network diagram, ignore the fact its all Ubiquiti gear, its the iconography I had loaded at the time. And I couldn’t quickly find icons for Netgear.

I also received a new to me MR33 complete with Meraki OS. Unfortunately their licencing of their hardware is not something that aligns with my principals on hardware so before we accept any software agreements I thought it best to do as always and flash OpenWRT to the thing and say goodbye to Meraki. Flashing OpenWRT onto it was no easy feat but I made sure the thing never saw access to the internet and eventually I was in.

[aidan@aidan-ld mr33]$ ssh [email protected]


BusyBox v1.28.3 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 18.06.1, r7258-5eb055306f
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:~# 
root@OpenWrt:~# exit
Connection to 192.168.1.1 closed.
[aidan@aidan-ld mr33]$

A couple of configuration changes later and we’ve got a dumb ap ready for deployment.

I really wanted to play with subnets this time, before we had two but this time I thought go hard or go home, so in total there are five subnets on my home network now, that’s enough IP space on the 192 network for 1200 devices. I’ve refrained from using the 10.0.0.0/16 network class simply because I use some VPNs with the same IP space so I thought best to just keep it simple. I have to say the Ubiquiti EdgeRouter X was a little bit new to me but I like the interface, PoE was a particularly nice touch.

For the Netgear switch the plan is to take a trunked VLAN ethernet cable and have the VLAN20 on the first 7 ports and leave the remaining 8th port for the MR18. Setting it up was quite easy using the online wizard however the visualisation thingy was clearly older than useful because it seemed to use some java applet that chrome did not like.

A Little Conclusion

I like what I’ve got set up now and will probably leave it for some time. The ZYXEL VMG8924-B10A is destined for the bin but the Netgear WNR3500L V2 I’ll keep for now, it can do VLANs and probably would have worked fine, I just wanted to use a managed switch.

The new network is great. I have a printer which its driver doesn’t seem to like cross-lan communication because every time I print something, it prints fine but the software client continually reports that the communication with the printer failed. I put that down to poor software testing I guess. Wouldn’t be the first time someone had printer woes.

EdgeRouter X Dashboard with VLANS — EdgeRouter Dashboard

Subnet	Purpose
192.168.1.0/24	Downstairs Wired (eth1)
192.168.2.0/24	Upstairs Wired (eth2.20)
192.168.3.0/24	Downstairs Wireless AP Clients (eth3)
192.168.4.0/24	Upstairs Wireless AP Clients (eth2.40)
192.168.5.0/24	4th port on router, only a printer attached.(eth4)

Subnets (eth0 is WAN)

Thats all folks. Aidan.

Scraping Canvas (LMS)

May 25th, 2020
Because my time at university is ending I thought it best to archive the canvas pages available to me for later reference should I not be able to access canvas later if they change platforms or disable my account. I should probably add this is for archival purposes and I will not be able to share the data I was able to collect. Thankfully I was able to get the whole thing going in a few minutes and downloading took a lot longer.

The first snippet I got from here, didn’t complete the first time, it seemed some image was causing issues so I moved to another gist, at this rate we could be done in half an hour 😊.

Unfortunately it also borked out on a similar place,
```
FileNotFoundError
```
I think it is because there’s something missing or I don’t have access to it. But the real problem is that its downloading content for a course I didn’t care about because I was enrolled in it but it’s full of junk I’m not interested in, so we can remove it by using the second scrapers code and specifying the course id’s which I had to manually go through, there was about 15 of them but it didn’t take too long. Which gave me the full command.
```
F:\Downloads\canvas>python canvas.py https://canvas.hull.ac.uk/ 4738~DUI9Nha9weSuemu1M2qsmhljoBcQtR0zghXTs3QA7ECHDHQkpsgBQ9RllbaEwySf output 52497,56148,56149,52493,54499,54452,54456,53441,52496,22257,22274,22276,22277,22278,22279,22280,50664,50656,22275,50652
```
The access token you can see above should be expired by now. You can do it yourself by downloading the same file and installing python3, pathvalidate and pycanvas. You need to generate a security token from /profile/settings and you can get the course id by clicking on the course like this /courses/56149. When you generate a new token you should receive an email about it.

Canvas online with our starred modules displated.

I decided to make a small adaptation to catch the FileNotFoundError and went off to the races. It took over an hour so I decided it was best to leave it running overnight, when I returned in the morning I had 116 errors (failed downloads) and the rest is the course content!

Our Canvas Modules saved to Windows File Explorer.

Unfortunately I don’t seem to have the submissions for each of these courses so I needed to manually download them aswell and then our archive was completed.

Thanks for reading.
How Much Does My Car Cost Per Mile?

February 15th, 2020

I thought its time I put my GNUCash Data to good use and worked out how much I spent on fuel. I loaded up a simple Cash Flow bar chart in GNUCash and selected my expenses column for Car>Petrol and voila.

Monthly petrol costs since 2017, highest at £167, lowest £10. Apparently I did not buy any petrol in July – Also checked this. Total expenses from 01/01/2017-01/01/2020 £2,659.79.

If you are astute you may have also noticed there are additional ‘fixed’ (varies annually/monthly) costs to running a car such as tax, insurance, maintenance and depreciation but I have chosen to eliminate these costs because I would like to explore the benefits of buying another car. We can use this graph based on the last 3 years to estimate this car, my 1998-1999 Vauxhall Corsa costs £73.88 monthly (excluding this month from the data)

My Car

So How far does £74/month get you?

Good Question, I’ve always used my car when I’ve needed it. I have little reason not to use it. I’ve used it to commute to University, School and Work and on days out. It’s my main mode of transport is what I’m trying to say.

Unfortunately, I don’t keep the history of my car’s odometer, however, I can use the mot history of my car to estimate the £/per mile. Using the mot history, which has the date my car was taken in for MOT for two different dates recorded in the MOT history, one in 2017 and one in 2019, we can determine in that time I did 19,205 miles so roughly 6.66k miles a year.

We can then use this 19,205 miles, which have around 1 year, 11 months between them to get (74*23) £1,702 expenditure during that time, which compared with the actual data gives £2,051.23 (over £300 diff, 21%) gives us a fairly low confidence, however we can use this to estimate my cars cost per mile on fuel alone is around £0.106 per mile. 10p per mile (2051.23/19205) or 740 miles per month, give or take 20%. That’s 24 miles per day!

Category: Projects

What Does a Scalable Application Mean?

What types of Scaling are there?

Best Practice for Scalable Applications

TLDR;

The Scenario

Making DDRESCUE Faster

DDRESCUE Results

Downloading from AWS S3 Glacier Deep Archive

Learnings and Final Thoughts

Version 0.0.3

The new network

A Little Conclusion

So How far does £74/month get you?