I decided it was a good time to learn docker and actually make a project that uses it, so I created ICO Security Trends, a small and unique dashboard which uses the UK ICO published spreadsheets to produce graphs and insight into the data.
I thought I would include some of my findings which are not immediately evident on the BI Dashboard they provide,
UK ICO Incident Security Trends
Categorisation on incidents described as ‘Other non-cyber incident’ has declined from 2019 to 2022. Roughly on average there are 750 incidents a quarter for ‘Other non-cyber incident[s]’, while ‘Other cyber-incidents’ remain fairly constant at around 60 a quarter.
The ‘Other non-cyber incident’ is generally too broad and should potentially be broken down. Insights into trends in this area are potentially being missed.
Ransomware disclosure has increased since 2019, which concides with general industry concensus.
There’s a lot more to it, but I thought I’d get it out there already,
Corporate Networks are highly thought out and well-designed critical business infrastructure that can span many buildings or geographies. The more complex an organisation is, the more expansive and multi-format the network can be.
A Corporate Network will often have an acceptable use policy and may monitor its usage.
D-Link DGS-108 Consumer Dumb Network Switch Corporate Network Server Closet
Features of a Corporate Network
Many corporate networks utilise additional benefits that home or small business routers usually are not capable of, such as;
Quality of Service or QoS is a layer 3 network technology that can prioritise (or more importantly de-prioritise) traffic by application, such as streaming, game services or file sharing.
Traffic Shaping is a bandwidth management tool to slow long running or high bandwidth downloads to prioritise other activities and ultimately restrict high utilisation on the network by a single client. This is most useful where bandwidth externally is limited.
VPNs (such as L2TP/IPSec or Wireguard) or SSL Tunnels (SSTP) allow corporate networks to link together across global infrastructure, SSL Tunnels can ensure that all data accessed by clients is encrypted by the link itself, so that any HTTP traffic for example must ultimately first travel SSL encrypted to the VPN routing appliance or provider.
VLANs can segregate and isolate riskier traffic as well as limit chatter or prevent sniffing ports. VLANs can also by separated by different subnets or network classes to protect, prioritise or isolate IT infrastructure and management from users. For example many switches have a management VLAN to prevent end-user clients re-configuring or accessing the management portal for the switch itself.
IPv6 is a relatively common new link format however some organisations are starting to implement IPv6 in their infrastructure in preparation for the switchover. Personally I believe this will not be a requirement for some time.
Content filtering and Proxying is used in organisations to protect valuable data and users from compromise and exfiltration. Some organisations require a proxy to reach external services and most implement some form of content filtering, generally for productivity or traffic management purposes.
DNS or Domain Name System servers can provide internal network resources resolvable and recognisable addressing for internal services. Most enterprises use DNS with Active directory through windows server domain controllers so that their Windows clients can take advantage of resolvable network names for windows machines.
Features of a Large Corporate Network
Larger Corporate Networks, ones that can encompass tens of thousands of devices or more could be considered large and may take additional setup, such as;
Load Balancing can be used to balance demand to external or internal services like internal enterprise applications or highly available applications that are business critical.
iBGP Routing or Border Gateway Protocol is usually only required for extremely large networks. Where routing and network policies are likely to change. BGP Routing is generally only required for carrier ISPs or enterprises dealing with internet infrastructure. For customers, due to the premium on network devices, the requirements of the networks used by enterprises and organisations are generally less than BGP can facilitate and BGP is not supported on smaller SOHO (Small Office/Home Office) networks.
Corporate Network Internal Services
DNS or Domain Name Systems
You may wonder how companies and other organisations are able to utilise top-level domain names that are not typically available on the internet, such as example.local and subdomains for a real domain, such as internal.example.com where internal.example.com is not a real external subdomain.
This is possible through many technologies and can incorporate many aspects to enable additional features like trusted SSL and network-level authentication or windows authentication to provide a relatively normal experience for end-users while being completely inaccessible from external networks.
SSL or Enterprise Trust
Even consumer routers often provide the facility to reserve DHCP addresses and register DNS names and aliases, but providing trusted SSL is accomplished through using either,
A local, trusted SSL certificate signing authority, with the organisations root or supplementary SSL certificate trusted by clients.
A real, actual trusted wildcard SSL certificate for a subdomain of the organisation. This is less common as it would require the same certificate to be on every application.
Network Segmentation and Isolation
A Corporate Network may utilise Network Segmentation to isolate external clients from internal applications or require a VPN to access. In this case, rules on the router allow inter-VLAN communication and routing table rules to allow communication with clients. Some networks may implement a zero-trust architecture in their network access.
Network segmentation restricts access to different services based on rules to help protect an enterprise from enumeration and the exfiltration of data, as access to the network is only possible through opaque rules that will make data transfer over the mediums allowed difficult. For example, access to a public server on a trusted LAN through a direct connection over SSH port 23 may not allow access to web-based interfaces internally such as port 80 or 443 as network rules prevent access, usually by dropping packets.
Many organisations may utilise these technologies in conjunction with an SSL proxy to provide legacy applications with an HTTPS frontend to a web server that is not configured for SSL, as access to the application web server would be restricted to only allow traffic through the proxy.
VPNs and DirectAccess
DirectAccess (similar to an always-on VPN) for Windows or VPN services like L2TP/IPSec enable corporate networks to be spanned over different environments, such as;
Field Engineers who rely on access to internal databases for parts or documents.
Mobile Devices and Tablets for reading email remotely.
Work from Home Deployments (WFH) for office employees who need access to shared drives and groupware.
Satellite or Remote Offices can deploy over the VPN to ensure a consistent experience for employees who travel.
Otherwise insecure environments, like coffee shops can be used as internal services will be accessed over the VPN and not directly over the internet.
Customer Premises where interfaces required on site can be relayed to internal networks at the origin organisation.
VPNs once configured with credentials can be utilised to provide network access as though they were direct clients of the VPN router, which could be placed in a trusted part of the enterprise and provide the typical trust, filtering and proxying required by the organisation configuration.
VPNs can often disconnect at work because there are packets not making it to the VPN provider. The simplest method to rectify this is usually by using an Ethernet cable.
Corporate Network IP Schemes
Unlike a public IP address with a single home network-attached, a corporate network may take advantage of using many IP addresses, networks and physical links to their ISP to provide a more robust and uniform experience to users.
Almost all corporate networks will use VLANs and network subnets to distribute their client environments to isolate services for example, a computer lab in a school vs a teacher network, or an open WiFi network at a restaurant compared to a private one for POS (Point of Sale) terminals.
Generally, most enterprises use the 10.0.0.0/8 IP CDIR block, using different subnets for different kinds of devices. Using the traditional 256 contiguous class C network addresses 192.168.0.0/16 range may not provide enough IP addresses for some larger deployments. (65,536 possible clients).
Corporate Network WiFi
Generally, Corporate Networks used to be a closed ecosystem, where only trusted devices and non-enterprise owned equipment was not present, this is no longer the case.
Rather than use combination Routing and Access Point devices like a home router, enterprises utilise extensive commercial WiFi Access Points that can provide access to numerous clients and can be distributed through the locations the organisation resides, like buildings and restaurants. Using dedicated hardware like Access Points enables the use of specialist configurations, like access point hopping for clients and PoE for easier installation and unification.
Some newer WiFi networks can also provide certificates that can be used in the organisation to access internal resources over SSL.
Applications that are depended on by thousands of users may see peaks or dips in demand during the day and managing the cost of running the infrastructure can be challenging. Scalable applications are applications that are able to increase their resources to serve requests as they come.
What types of Scaling are there?
There are two basic examples of scaling for applications,
Vertical Scaling, where an application’s resources are increased due to demand, such as increasing the RAM available to an application host. Vertical Scaling is also sometimes referred to as scale-up.
Horizontal Scaling, where an application is spread across different nodes in a cluster. This is most appropriate for applications that require a lot of resources. Horizontal Scaling is also referred to as scale-out.
Scalable applications have many uses, including;
Allowing cost reduction during low utilisation by scaling down clusters or pods serving the application.
Improving quality of service during peak load times by scaling up horizontally or vertically by utilising resources made available to the application autoscaler.This will ensure that your application always has the right amount of capacity to handle the current traffic demand.
Faster processing as features can use the optimal storage technology for data and information.
Best Practice for Scalable Applications
Applications usually are best compartmentalised into components for both design and maintainability, monolithic architectures for code bases and applications have caused application elements to become obscure and entrenched, using a more distributed approach from the start can reduce cost and technical debt as components can be re-written or swapped out.
Splitting transient application functions in to their own libraries or APIs can allow the front end and back end to operate independently, this can allow processes that take time to be acted on based on events rather than cause waiting or processing as a batch.
Data storage should be independent to the service that owns the data and therefore should use the best storage type and mechanisms available such as caching or streams of data. Slow returns should be accounted for and independent from the user interface.
Applications should not be strained by resources, whenever possible applications should perform the same for every user or function regardless of workload or work-type. Rather than wait for functions to complete, have a common goal of eventual data integrity.
When implementing concepts like micro-services you should endeavour to ensure standard practices like a common set of languages or behaviours to improve maintainability.
Complexity can sometimes be harder to manage than an equivalent monolithic application, even though each function should be simpler.
I recovered my data from AWS S3 and all I got was this lousy bill.
Aidan – Alternate Headline.
TLDR;
One of my hard drives failed, I thought I’d try to recover the valuable 400GB using ddrescue, it sort of worked.
Restoring from S3 is expensive £27.53 for ~400GB
The Scenario
A week or so ago I realised that my hard drive was on the way out, its been on for almost 27,000 hours according to the SMART data. I first noticed when the PC was loading into check disk after every reboot. It took me about 3 reboots to decide something was up and I used Crystal Disk Mark to check the disk and sure enough it was reporting ‘Bad’. So I ordered 2*6TB drives and thought I’d better have a go at moving the data and making sure my backups were up to date.
For my backups, I use cloudberry backup (now called something else) which is an encryptable cloud backup solution which is compatible with Amazon’s S3. I use the cheapest storage option, S3 Glacier Deep Archive.
DDRESCUE GUI Data RecoveryDDRESCUEVIEW Preview
I booted in to a persistent live Ubuntu 20 environment and installed ddrescue, ddrescueview and ddrescue-gui. I found that the tools worked well but took way to long for the drive, you can see in the remaining time section of ddrescue-gui it would have taken an estimated 60 days to recover the data at the fastest setting.
Making DDRESCUE Faster
To make ddrescue faster I found it was best to watch the drive speed in ddrescue-gui and then I scrapped it over the command line for a faster experience.
In the end I used these commands, make sure to replace the drives with your setup and the minimum read rate to one your drive is comfortable with. For the first command, I stopped it at around 90 percent of the way through the drive and swapped it for the second one.
# First run to cover myself in case the drive died more seriously.
sudo ddrescue -f --reopen-on-error --min-read-rate=8000 /dev/sdd2 /dev/sdc1 /home/ubuntu/Documents/log1.log
# Lots of Passes to try to recover slow sections.
sudo ddrescue -f --reopen-on-error --retry-passes=5 /dev/sdd2 /dev/sdc1 /home/ubuntu/Documents/log1.log
Although this really seems like a your mileage may vary environment depending on the type of failure your drive has.
If you do end up using ddrescue-gui at least to begin with, you can use the log file to get you a command to start off with. Make sure to read the manual pages for ddrescue to determine the best command for you.
Here is an example of one of my outputs (.log files),
You can of course view this data using ddrescueview.
ddrescueview
DDRESCUE Results
After a week and a bit, I decided to stop the experiment and see what had been recovered. ddrescueview looked like this,
ddrescueview results
ddrescue was able to recover about 90.83% of the ntfs partition, enough to mount the drive and view the data. It contained many of my important personal files and more importantly images and home video. The actual used space on the drive was only ~700GB, with around ~450GB of data that was valuable to me.
When I opened the personal photos and videos, I found the results to be quite poor, there were glitches in them, sometimes files had no actual data in them, sometimes they had stripes and lines in the image, because of the spread of the failure across the partition blocks, the data was basically a really poor copy with a lot of holes.
ddrescue non-tried sectors made many files unreadable or poor quality
I decided that it was best not to continue the recovery with ddrescue and instead restore from backup, the age of the backup was exactly 1 month prior to the failure to the day, so no real loss. However only the data that I truly cared about was backed up. So stuff like my VMware ISO files and downloads folder were lost and unrecoverable.
Downloading from AWS S3 Glacier Deep Archive
Using Cloudberry I made a restore plan and recovered the data using the slowest SLA at 3-5 days, which by sods law took the full amount of time to process and then some, because I put in the wrong decryption password and needed to re-request the data.
Anyway here is the bill, £27.53
AWS S3 $37.31 (£27.53) Bill
The killer was the data transfer fees out of London, at a cost of $0.09/GB ($28.37).
And with that, all of my data was re-recovered, this time without corruption.
Learnings and Final Thoughts
Although AWS S3 is a valid backup option, its expensive to recover from. I already pay roughly $1/mo for ~400GB (315GB compressed). For larger recovery this would be prohibitively expensive, multi-terabyte or whole disk backups would require compression.
Physical damage to a hard drive is essentially game over, your data is lost. For best results have redundancy. This is the only reason I am thankful for S3, it was my only solution to recover my data. A local backup would have been much cheaper and faster to recover.
The two new 6TB drives run in a Windows storage spaces two-way mirror pool.
Its been a good while now and Flashcard Club is well on its way to a functional product. There has been some progress on features, notably the inclusion of Google OAuth2 through Laravel Socialite.
Logging in through your Google Account should greatly increase the speed of adoption for new users and improve retention and user acquisition.
With that in mind, Flashcard Club has been online now for about 3 months and has yet to have a single user. I believe this is mostly due to myself not promoting the product which I will feel more comfortable doing once the site is ready, however it is currently somewhat usable. To that I have made improvements in the homepage, mostly bringing it up to paces with a call to action and have moved the changelog to an FAQ page, which I may change later as I do not love the name.
Users can now login and link their Google Account to Flashcard Club.
Landing Page
The Landing page has had a massive makeover and most of the content is different now.
The changelog has been moved to the FAQ page.
Test and Study Mode
There is now a chart to plot test performance per set.
Test mode now has additional functionality like gold highlight on completion of the test.
Test Mode now has a summary.
Planned Features
Google Sign in (Federated identity) Completed
Terms of service
Privacy Policy
FAQ Page Somewhat Completed
Markdown User Guide
Flashcard User Guide
Front page needs work Not complete but looks a lot better.
Cramming mode that removes cards previously marked “Correct”.
Part of Improving the site in the next round will also be improvements to the mobile aspects of the site as most users will likely be on mobile devices.
I have also been ignoring the fact there is currently no export option available to users.
It’s been a little while since the 25th of December when I decided to go public with this idea, a great deal of work has gone into flashcard club. Exactly 40 hours has now been dedicated to the project outside my normal 8:30-5.
It’s been quite slow going in some aspects, one particular problem is the markdown library options are numerus and vary in completion and schema that make it hard to choose one that is appropriate for this project. In the end I have settled on marked.
Another area I am struggling to decide on is how best to onboard new users, I want them to have a positive experience however not all know the markdown syntax.
Currently, Flashcard Club has the following features,
User Accounts,
It’s currently possible to create a new user account, although there is no ability to ammend or update your user account.
New users should recieve a welcome email.
New users should be logged in automatically.
New users should be able to create flashcards and flashcard sets.
Almost Unlimited Flashcards,
It is possible to create and save flashcards.
You can also update flashcards, delete them and more.
Flascards use markdown syntax so you can include pictures and a limited amount of html.
Unit Tests,
Many components of Flashcard Club have PHPunit Unit tests.
Code coverage is probably very poor.
Toast Notifications,
Toast notifications are intergrated into the website.
The best place to see these in action is in the set editor if you update the title.
Planned features include,
A Study mode, because currently it is only possible to edit flashcards but not view them without the editor.
Google Sign in (Federated identity).
Statistics, because who doesn’t love a good graph.
Account management, because currently once you make an account you cant change any of your details, oops.
I’m sure there is not many of you but I had to tell someone. As of the last few days I’ve really felt the need to get involved in something that is really what I am all about. A new baby if you will, a commercial enterprise but one I own. I’ve been working full-time on Laravel projects and I want to improve my skills to really get better at it. Its a great framework after using raw PHP and template engines like handlebars.
During this time I came to the realisation that I wasn’t totally stoked with the existing flashcard companies out there and I think now more than ever I really have the skills to get something like this off the ground, hopefully this is the first project of many!
Flashcard Club is different,
I really want a modern material feel to the website. I’m not interested in something worse than what there already is out there, I want to really help people.
I want to really build a solid brand that people want to use. Flashcard Club should be a place to really get some work done and cram for that final test and study in a safe space.
I really want to provide a platform for students to get some work done. A lot of the competition has low quality submissions. I really want people to have an effective space to learn.
I want to publicise what I’m up to, and this blog will help with that. A real leading learner approach. From conception to reality.
Although this will have options to buy products, long term I only want this project as a sort of demo of my capabilities. I want to move on to better ideas ASAP but you have to start somewhere.
This is my first “premium domain” project so consider that an invested interest in this project.
Have a look and see how I’m getting on. At the time of writing cloud flare gives out error 526 so I hope to be past that by the time this goes up a month from now.
I aim to make both a website and mobile app, however the majority of the features will be made through the app, the functionality and backend will be possible through the online version as well.
I think my target audience is 99% students so a strong app presence is a must I feel. The website will be a strong start but is just one part of the experience.
I used geocraft to generate a Minecraft map of the University of Hull, and now I am sharing it with you. You can see a short video of the map below.
The map is available as OpenStreetMap data only, or with the DEFRA (Department for Environment, Food & Rural Affairs) lidar data from, (I think) 2015, so the multi-story car-park is not included for example.
The original project is broken as the lidar data did not get included, but it was possible to fix it using the suggested changes by RobinWhitfield here in this issue.
The original licencing for this map is CC-BY-SA for the pure OpenStreetMap map and the other is the open government licence. You may use these maps as you wish within these constraints.
If you plan on running these maps on a server, you may want to turn off leaf decay as some of the trees are all leaves and do not have a wood block to prevent them from decaying, as seen in the video.
I recently have been working on a weather recording project and as part of this endeavour wanted to check that all of the variables I have posted to my API were set, it can be rather a lot of work to have to specify each if (!isset($_GET[])) parameter so I decided to use a function that can take any number of arguments using a ‘variadic’ function in PHP.
// Returns true if all variables are set, else returns false.
function getVariablesSet(string ...$getlinks) {
foreach ( $getlinks as $link ) {
if (!isset($_GET[$link]))
{
return false;
}
}
return true;
}
Now, when I want to use a new api ‘command’, I can simply do the following,
$browser_response = new stdClass();
$browser_response->message = "Command not specified.";
if ($_GET['command'] == "new-temperature")
{
if (getVariablesSet("datetime","temperature","humidity"))
{
$browser_response->message = "All GET variables set.";
}
else
{
$browser_response->message = "All GET variables not set.";
}
}
echo json_encode($browser_response);
This way, so long as we have specified the command and the variables required we can enter the scope, otherwise, we can kick them out until their query is formatted correctly. Having lots of variables may become problematic so you may want to use POST or even break them out into subsections to give users a better understanding of their error. I should add this only works for PHP 5.6 and above.
Hello again, this is the second in a series of my home network posts. In this edition, we’re going to be flashing another router with OpenWrt. You can read part one here.
I decided it was time to leave all the old hardware behind and move to brand new stuff. I was having problems with the ZYXEL VMG8924-B10A as the main router, it kept cutting out and was causing short minute outages that ultimately I think was due to the system running out of ram and botnets trying to break into the thing. So I decided it was time to jump ship and move to something a little bit more enterprise. At the same time, I decided now would also be a good time to leave the Netgear WNR3500L V2 to one side. Despite it serving us good for many years, its routing features won’t be necessary for the upgrade as I want to have a good crack at managed switching.
The new network
The new network is composed with the following,
A Ubiquiti EdgeRouter X
A Netgear ProSafe GS108T
A new to me MR33
I wanted to do away with having two routers on the network, it wasn’t neccisary and caused some of the upstairs equipment to be inaccessible from downstairs, the new configuration would mean that all management would be done through the EdgeRouter X which would be much cleaner and hopefully faster.
The ProSafe GS108T was a garage find, I wasn’t using it for anything and I hadn’t really implemented proper VLAN tagging before so I thought now was a better time than ever to get into it.
My new home network diagram, ignore the fact its all Ubiquiti gear, its the iconography I had loaded at the time. And I couldn’t quickly find icons for Netgear.
I also received a new to me MR33 complete with Meraki OS. Unfortunately their licencing of their hardware is not something that aligns with my principals on hardware so before we accept any software agreements I thought it best to do as always and flash OpenWRT to the thing and say goodbye to Meraki. Flashing OpenWRT onto it was no easy feat but I made sure the thing never saw access to the internet and eventually I was in.
[aidan@aidan-ld mr33]$ ssh [email protected]
BusyBox v1.28.3 () built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 18.06.1, r7258-5eb055306f
-----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:~#
root@OpenWrt:~# exit
Connection to 192.168.1.1 closed.
[aidan@aidan-ld mr33]$
A couple of configuration changes later and we’ve got a dumb ap ready for deployment.
I really wanted to play with subnets this time, before we had two but this time I thought go hard or go home, so in total there are five subnets on my home network now, that’s enough IP space on the 192 network for 1200 devices. I’ve refrained from using the 10.0.0.0/16 network class simply because I use some VPNs with the same IP space so I thought best to just keep it simple. I have to say the Ubiquiti EdgeRouter X was a little bit new to me but I like the interface, PoE was a particularly nice touch.
For the Netgear switch the plan is to take a trunked VLAN ethernet cable and have the VLAN20 on the first 7 ports and leave the remaining 8th port for the MR18. Setting it up was quite easy using the online wizard however the visualisation thingy was clearly older than useful because it seemed to use some java applet that chrome did not like.
A Little Conclusion
I like what I’ve got set up now and will probably leave it for some time. The ZYXEL VMG8924-B10A is destined for the bin but the Netgear WNR3500L V2 I’ll keep for now, it can do VLANs and probably would have worked fine, I just wanted to use a managed switch.
The new network is great. I have a printer which its driver doesn’t seem to like cross-lan communication because every time I print something, it prints fine but the software client continually reports that the communication with the printer failed. I put that down to poor software testing I guess. Wouldn’t be the first time someone had printer woes.
EdgeRouter Dashboard
Subnet
Purpose
192.168.1.0/24
Downstairs Wired (eth1)
192.168.2.0/24
Upstairs Wired (eth2.20)
192.168.3.0/24
Downstairs Wireless AP Clients (eth3)
192.168.4.0/24
Upstairs Wireless AP Clients (eth2.40)
192.168.5.0/24
4th port on router, only a printer attached.(eth4)
