At the beginning of November, Monologue was a macOS app loved by thousands of people for its ability to save time by dictating instead of typing and its unique skeuomorphic look.

Naveen, the general manager of Monologue, was asked by his users to build an iOS version and vibe-coded the first prototype in a single week in November. He did this to figure out how Monologue on iOS could work and to prove the concept.

Shortly after, we bumped into each other on Twitter and hit it off over a shared appreciation for each other's work. He asked me if I wanted to help him build Monologue for iOS, and I was ready for the challenge. He told me he had already built an early prototype and got me to use it on the same day. A week later in December, I started working together with him and Daniel to bring his vision to life.

The Prototype

When I first opened Monologue's codebase in December, I had to decide whether to work with Naveen's vibe-coded prototype or start from scratch.

The version he built with Codex used the macOS app as inspiration and got him impressively far. All the core features were already in place and hadn't changed much since.

The codebase had the usual vibe-code smell: code duplication, misuse of Swift 6 features like @Observation and Swift Concurrency, and poor integration of third-party dependencies. There were dozens of compiler warnings, and it was impossible to move fast because dependencies between modules were all over the place.

Instead of starting from scratch, I decided to keep working with what we had to tighten the feedback loop between us and Monologue's users. Right from the beginning, we made sure to share new TestFlight builds with the team daily and weekly with interested users from Every's Discord.

The Foundation

I spent the first dozen hours cleaning up the vibe-coded core to be able to move faster and reuse more code. The plan was, right from the beginning, to modularize code as much as possible to encourage reuse between the iOS and macOS apps.

Once the foundation was solid, I started to work on one big feature after another in tandem with Daniel, who provided me with amazing new designs every couple of days. Naveen, Daniel, and I met every other day in a conference call to share progress and discuss Daniel's latest designs. We worked fully remotely, with Naveen being in India and NYC, Daniel in Panama, and me in Indonesia.

Whenever I started to work on new designs, I completely ripped out the vibe-coded mess and replaced it with a feature module with a clear dependency graph. This process continued feature by feature until almost no vibe-code was left.

During this process, Naveen and I started to modularize core business logic for code sharing between the Mac and iOS versions. We also started to work on a shared MonologueUI library that both versions now depend on.

Design Iterations

Monologue's skeuomorphic design is as challenging to implement as it is beautiful. I feel lucky to have a wonderful team that cares as much for every little detail as I do. Daniel was patient, and Naveen kept pushing for more polish.

When working on intricate designs and polishing details, fast iteration is everything. The initial keyboard prototype was horrible to work with in this regard. To see a change, we needed to recompile the app, open another app to present the keyboard, and then test our changes.

Monologue's keyboard has too many states for this to be a joyful development experience. I rebuilt the keyboard from scratch, completely decoupled the UI layer from its business logic, and built a SwiftUI playground to instantly see changes while coding.

To bring Daniel's designs to life, I used SwiftUI's animations to make buttons feel more realistic. Instead of using image assets for buttons and UI controls, every tiny detail is implemented using SwiftUI. This lets us create more realistic state transitions, like changing shadows while tapping a button or shiny display effects.

Conclusion

Naveen vibe-coded an extensive prototype to find out what Monologue on iOS could feel like. He validated his idea and got everyone at Every excited. Early users were happy to have Monologue in their pocket, even though it didn't feel as polished as the Mac app yet.

He never intended to ship the prototype to a broader audience. He proved the concept and knew this would be a great app if he got the right people involved. He brought in Daniel and me to turn his vision into a polished product — modularizing the codebase, rebuilding the keyboard from scratch, and sweating every detail of Daniel's designs.

This is the perfect use case for vibe-coding. Not as a shortcut to a finished product, but as a way to validate an idea fast and then hand it off to people who care about the craft.

If you want to read more about »How to Design Software With Weight«, you can read the article Daniel and I published over at Every.

Using the App Store comes with multiple benefits like automatic distribution of updates, TestFlight, a merchant of record payment provider via Apple's StoreKit, and sometimes visibility. Those benefits come at the cost of 30% of your revenue.

But the Mac App Store also puts many restrictions on developers through Apple's strict App Store guidelines. Not every app can conform to Apple's guidelines, like using the sandbox entitlement, which strongly limits what an app can and cannot do.

If your app cannot make use of the sandbox entitlement or doesn't conform to the review guidelines, your only option is to distribute the app yourself. That means you will need to manage app updates and payments yourself. For payments, you most likely will need to implement some kind of licensing system. To keep your app up-to-date, you will use the brilliant Sparkle framework. But setting both of these up is cumbersome and takes time away from building your app.

Sparkle

The Sparkle updater project is the de facto standard to distribute app updates outside the Mac App Store. It's decades old, reliable, and familiar to users. But setting it up will consume your day. The process of publishing a new update to your app using Sparkle also takes a bit of time. I assume that it costs me around 30 minutes each time I want to ship a new update for my Mac apps.

The process for each update:

1. Archive the app
2. Notarize the app
3. Staple the notarization ticket onto the app
4. Create a DMG or ZIP the app
5. Sign the update via Sparkle
6. Create a new appcast.xml
7. Manually edit the changelog in the appcast.xml
8. Upload the resulting files to some server

If you want to use some of Sparkle's more advanced features, like minimum system version requirements, major or critical updates, phased group rollouts, or beta channels, you will need to manually set that up as well.

My Vision

What if there is a project that has all the benefits of the Mac App Store without having the downsides like App Store Review and strict sandboxing requirements plus the 30% Apple tax?

That's what I am planning to build. My vision is to offer a simple developer toolkit that makes distribution and licensing of macOS apps easier and faster than publishing an app on the Mac App Store.

I split the project into 3 phases:

1. Research and feedback
2. Managing of app updates and distribution
3. Payments and licensing

Managing App Updates and Distribution

This project is still in phase 1, but for my research I started to tinker around with a command-line tool I call amore.

The end goal of amore is to have a single command to release a new app update. Whenever you want to release a new version of your app, you simply execute amore release and it will take care of the rest for you. From archiving your app, through notarization, and uploading it to a server. All the 8 steps from above. amore will be able to take care of it.

On your app's side, you will only have to add 2 new Info.plist entries to set up Sparkle. A URL for the appcast.xml so your app knows where it can check for new updates and your public ed25519 key. Don't worry, amore will handle key pair generation and management for you, too.

So far it can only do steps 4 through 8, but as I continue, I will add all the remaining steps. It's a single binary that's fully compatible with Sparkle and has 0 dependencies.

amore pairs with a server to manage updates and advanced Sparkle features like optional anonymous analytics collection about your users and more. In addition I want to offer a web- or app-based WYSIWYG editor to make the creation of release notes easier.

In addition to the command line tool, I am thinking about a Mac app that you can just drag updates into and manage new releases. Although I am not sure if developers would prefer this instead of a CLI.

The app could work like Helm, but without being slow and built around App Store Review.

Purchases & Licensing

This is a bigger project and will probably have to happen in stages because I need to learn more about use-cases and what developers actually need.

For payments, I would start with a bring-your-own-API-keys model compatible with Stripe, Gumroad, Paddle, etc. Eventually I would like to provide a merchant of record scheme to be able to offer a full package, but getting there will take time.

The plan here was to start with the 2 most popular licensing models for your apps:

• One-time purchase
• One-time purchase with free updates for a year

This would come with a client SDK to make implementing license validation straightforward.

Security & Privacy

App updates are a critical failure point in your application, and that's why the service won't require your Apple credentials or your private ed25519 key used to sign each Sparkle update. All the steps requiring private credentials or keys will happen on your computer with your permission.

To stay independent of amore and the server you will be able to use your own domain, preventing vendor lock-in.

Pricing

To finance the service we need to charge money. I don't know what a realistic price is yet, but for the update distribution I was planning to charge the same as Apple. An annual fee of $100 for unlimited apps. Free for FOSS projects.

I am not sure how to monetize the payments and licensing part yet.

Your Feedback

For now I'd like to learn more about the Mac developer community's wants and needs. The purpose of this post is to let you know what I envision and to spark feedback and discussion.

• Is this something you would like to use?
• What do you need?
• What's your biggest pain point when distributing apps outside of the App Store?
• Would you pay for this?

Please share your thoughts with me via this feedback form, email or Mastodon.

If you are interested in the project and want to get notified about updates, you can use this form to give me your email address. I won't spam you. I promise.

Inspiration

This project heavily inspired by Zhenyi Tan's Idea: A Modern Potion Store and my own frustration with how Apple governs the Mac App Store.

Thanks for your ongoing advice and guidance, Nim

A few days after we released version 1.0 on the Mac App Store, we got featured on the 'Discover' page, which brought tens of millions of impressions and a flood of new users. At that moment, I was in Germany preparing my brother's wedding and didn't have the opportunity to work on the issues all the new users were reporting. And there were plenty.

The most annoying issue for most was that when using Macrowave, the system would automatically lower the system volume on Macs (a behavior called ducking), making it less fun to share what you are listening to because it would worsen your own listening experience. Another issue occurred when people were accessing their Macs remotely and started a broadcast. MacOS would crash our app because it also used the screen sharing API to get system audio.

Our Vision

When Neco and I started to work on Macrowave we had a strong vision of how the macOS app should look and feel. While other internet radio setups require technical skills to configure and set up an Icecast server or similar, we wanted to have a solution that provides everything out of the box to get started. It should be a simple app that everyone could use, with a single button to go on air, and once activated, it would just fade into the background and do its thing without impairing your Mac's performance.

We were unable to execute that vision in the way we wanted to because of some initial limitations of our upstream provider, LiveKit. Eventually we reached out and asked them for help. They came through a couple of weeks after the launch and updated their SDK for us.

New Engine

With all the required pieces in place, I got to work in early October and started to rewrite our audio engine from scratch. Instead of having to rely on LiveKit's SDK to get system audio via the screen recording API and use their sub-optimal implementation to get audio from a microphone, we were free to build what we needed.

It was the first time I worked directly with audio frameworks on macOS, so it took a while, and I learned a lot.

To get system audio, we now use Core Audio Taps, which means we can tap into audio from specific apps directly without having to ask to record parts of the screen, which felt invasive and unnecessary. This comes with an added performance benefit, as the video part of the recording was always dismissed.

The resulting engine doesn't require screen recording permissions, consumes fewer resources, and allows to have multiple app audio sources. Additionally, we now have more fine-grained control over each audio input, which, for example, means that in a future update you will be able to control the input volume for each connected microphone and app separately.

Minor Improvements

In addition to the audio engine, we also improved some smaller aspects around the app. We got rid of the flipping animation when switching between broadcaster and receiver that didn't really work on macOS Tahoe and replaced it with a sliding animation. This comes with the added benefit that we were able to add the window border and shadow users were missing.

We also added a minimal troubleshooting screen to help users fix their permissions in case they accidentally disabled the microphone or audio recording permission.

You can find the full list of changes in our Kinopio Space.

Demo

As an experiment, I recorded a quick video showcasing the benefits of the new audio engine. I can see myself doing this more often if this is something you like. So please reach out to me.

Discord

Because of the increasing amount of broadcasters and users that ask for stations they can listen to we created a discord community that you can join here:

Join Discord →

Two years ago Today, I wrote my first morning pages. A practice I wouldn't want to live without anymore. To celebrate the anniversary, I planned to launch the iPad version of Pagi on the App Store today.

I submitted it for review on Thursday 5 days ago, to give the review team enough time to point out things I have to fix, in case I missed any of their guidelines.

On Friday morning I found my submission rejected with the following message citing guideline 4.3 Design: Spam of the App Store Review Guidelines:

We noticed that your app provides the same feature set as other apps submitted to the App Store. Specifically, this app appears to be similar to another app previously submitted under a terminated Apple Developer Program account. The next submission of this app may require a longer review time.

I can see how Pagi is similar to other apps in the App Store as it features a full screen text editor, if you dismiss its unique features designed for the morning pages use-case.

The claim that it 'appears to be similar to another app previously submitted under a terminated Apple Developer Program account' doesn't make sense to me. 'Terminated' also means that the previously submitted app is not on the App Store anymore. So even if it did appear similar, it shouldn't be a problem and, by definition, can’t be a duplicate.

I replied and explained that Pagi is not just another text editor, but specifically caters to the use-case of writing daily morning pages. I listed the following unique features:

• Disabled spell-checking for distraction-less writing of a constant stream of consciousness
• Auto-clearing of the written text on the next calendar day.
• A word count and progress bar to indicate how far the exercise is completed.
• Customization options with multiple themes and fonts.
• Focus Mode to focus on the currently written sentence or paragraph

Additionally, I told them that Pagi already has a user-base on macOS and that users asked for an iPad version. Furthermore, I asked:

• Is it possible to get more information on why you rejected this app?
• Which app in the App Store provides the same features set as Pagi?
• To which app does Pagi appear to be similar that was previously submitted under a terminated Apple Developer Program account?

To which they replied a day later:

Thank you for your reply. Just as we would not share information from your Apple Developer Program account with another developer, we do not share the details of apps submitted under other Apple Developer Program accounts.

Which is a weird excuse. They claim that my app is too similar to another App that is no longer in the App Store, but wouldn't even tell me which one. It's a reply that doesn't answer any of my questions.

They continued:

During our review, we found that this app duplicates the content and functionality of other apps submitted to the App Store, which is considered a form of spam and not appropriate for the App Store.

They wouldn't get more specific than this, which is frustrating. Especially when you consider all the duplicated apps you can find in their store when searching for terms like 'flashlight' or 'fart'. While the term 'morning pages' will only show you 8 results from which only 2 are dedicated to morning pages. Both of them are inferior to Pagi's design and don't offer the same set of features. Neither of them seems maintained.

They continued with a final punch in the face, by citing the 'Develop' section of their developer website.

Apps submitted to the App Store should be unique and should not duplicate other apps. We encourage you to create a unique app to submit to the App Store. For more information about developing apps for the App Store, visit the Develop section of the Apple Developer website.

Neither of the things they told me are helpful. They don't give direction on what to change or improve to get Pagi approved. Instead, they told me to abandon the entire project and start from scratch with another app. Completely dismissing user demand and all the time it took to build this app.

As a cherry on top, they removed the version I submitted from TestFlight, which is the last nail in the coffin for Pagi on iPad. Now I have no way to distribute Pagi for iPad to my users.

It's public knowledge, that the App Store Review team is doing a horrible job at policing the quality of submitted apps. The last, more prominent example that comes to mind was the brilliant Mastodon client Ice Cubes, which also received weird and opaque reasons for its rejection. It got approved after the linked blog post on Daring Fireball was published. It shouldn’t be like this.

I feel discouraged spending more time developing apps for Apple platforms, when there is no guarantee that I can distribute my creations. It takes a lot of time to get an app into a state where it's ready for review. Besides the actual coding, Apple requires developers to work on marketing materials like screenshots, copy, and a website before submitting the app. It’s even more frustrating considering all the trash apps that they not only offer, but advertise in their store today.

I advocated for a way to side-load apps on iOS devices before this happened to me, but now that I went through this experience, the need for it became even clearer. I hope regulators around the world will start to act on opaque gatekeepers like Apple soon.

Update: 2023-03-01

I don't understand how it happened yet, but I woke up to the app being accepted, without any further notice.

Thank you all for sharing this blog post and the words of encouragement I received during the last 24 hours. I am thankful to every single one of you.

You can find more information and a link to the App Store on Pagi's website.

Update: 2023-03-02

The App Store Review team called me yesterday to apologize and to clarify the situation.

When they received my app, they noticed that the code as well as the app icon was used by another third-party developer, without realizing that I am the original author. They guess that this might have happened because I was developing Pagi publicly on Futureland and GitHub, so the source code and assets were publicly available before I submitted it. Fair enough, they did not tell me any information about the copycat developer, except that their developer account got terminated.

The review team initially upheld the rejection, because the information of evidence they found on their side was very obvious. The case eventually got escalated internally, and they were able to verify that I was the original author of the app and accepted my submission.

In case I have the feeling I am in a situation like this again, I should submit an appeal to App Review.

After everything that happened, I am impressed how quickly they acted after they verified that I am the original author. They called me on the same day to apologize and clarify the situation. I appreciate that.

I think it's a good thing Apple has this process of checking for duplicates to identify bad actors in the App Store. This definitely serves developers, but their communication could have been better. They should have pointed out ways to verify my authenticity instead of the vague messages they sent me. When I read those messages again now, I understand what the issue was and how I could have solved it, but 48 hours ago it didn't make any sense to me and to a lot of other developers that reached out to me.

I am happy that this is resolved and Pagi is in the App Store. By reaching out to me and explaining the whole situation, the App Store Review team was able to regain my trust.

In this blog post I will explain why I rebuilt lucas.love and how I tried to improve it.

Why?

I haven't published anything to my blog since almost a year. Before I was publishing multiple blog posts every month, but then I decided to drop the Ghost backend because I did not want to have my blog's contents inside a database. While this sounded like a nice idea it increased the friction to publish anything new to my blog, which eventually led me to never publish anything again. The problem was, that for every update I wanted to make on the site I had to boot up my development environment, make changes and then commit to git. It was just no fun to use anymore.

The Solution

I decided that I want to use markdown as markup language as it restricts me, but not too much. It also makes it very easy to just throw some images into blog posts. I set it up in a way where I can link images using a relative path like image.png and the image will automatically get rendered. This makes writing new blog posts a delight.

As content management system I wanted something really simple. Something I can use to compose new blog posts even when I am offline. I got inspired by blot.im and the idea to use Dropbox (or a similar cloud service) and plain text files as CMS. Usually I am not a big friend of using third-party cloud services but in this case it would only store files that are public anyway, so I don't mind.

I like this solution because now I can make quick changes and adjustments to contents in a local folder, that is easy to backup. I especially like the longevity aspect of using a folder of just images and markdown files as CMS. I will still be able to view the contents of this folder long after my website is gone because this approach does not make me dependent on any third party tool. Even if Dropbox ceases to exist I still have all of my data and can find an easy alternative.

This approach also allows me to use iA Writer on my iPad while I am offline or not at my desk. Once I am online again I just have to let Dropbox sync and trigger a new deploy on Vercel.

The website itself is built with Nuxt and the @nuxt/content module. During build time I download the content folder from my Dropbox via a public share link and unzip its contents into the content folder of the Nuxt project. In the next step Nuxt generates a static version of my website and Vercel will deploy it to lucas.love

To trigger a new deploy after I change some files in my Dropbox I use a simple Apple shortcut to make that process easier.

The Result

In addition to the CMS changes I wanted to make some improvements to the design. I want to add more different kinds of contents like a knowledge base and recipes later. I knew that I had to improve the navigation because some users did not know how to go back to the index page to select another blog post or another project. So I thought it would be neat to have the navigation always present on the left. Additionally I recently saw a website that used a 3 column layout, which I liked. I am using this for the blog section of the site.

Additionally I tweaked the typography and chose a new primary color and made the decision to make English the primary language instead of German.

Before

After

I document the progress of working on this website on Futureland.

I never counted how much I drank but my family and friends made jokes about me and told me that it is unusual to drink this much water. I don't know if it's true. I just like to be hydrated well and cannot think when feeling thirsty. I know I am pretty boring but I usually don't drink tea or coffee. I just like to drink water. And yes, it can taste quite different. My favorite mineral water is Elisabethen Quelle from Germany.

Experiment

One of the most used journals on Futureland is the water journal by Vin. It is described as:

Internet friends trying to drink more water together. Drop an emoji when ya drink a glass! Your stats will show you how much water ya drink.

I think it is a fantastic idea and cool Futureland use-case but as I never had problems staying hydrated it never appealed to contribute.

In May I played around with Siri shortcuts for the Futureland iOS app and released a Futureland shortcuts integration a few days later. Since then I started to use the shortcuts integration for a bunch of my journals as they provide faster ways to publish. I got curious how I could use all of this to track my water intake.

I live in Indonesia right now and 6-packs of 1,5 liter bottles are not available. People usually buy a gallon of water and use a cooled dispenser for their hydration needs. Because for water I prefer to drink out of bottles I bought myself a 500 ml stainless steel hydration bottle. While working I usually have to refill it at least once an hour.

Shortcut

To make tracking of my water intake easier I created the following iOS shortcut.

First it creates a new entry on the water journal with the content 500 ml. Afterwards it logs a new health sample into Apple Health for the statistics.

Tracking

I wanted to keep the amount of work for this experiment as low as possible. I am pretty pleased with the result. The only thing I have to do is to press one button on my Apple Watch every time I my bottle is empty. There is no menu, no waiting time, nothing to keep in mind. By now it became a habit. You can see the shortcut button in the bottom right corner of this screenshot.

Results

In general Apple Health provides a nice way to look at all kinds of health data. For water this is not really exciting but it does the job. Additionally it shows a bunch of useless data like

You drank water on 7 of the last 7 days.

and

You drank water on 28 days in the last 4 weeks.

In the last 4 weeks I drank on average 3,3 liters every day. This means on a usual day I refill my bottle 6-7 times. A gallon of water in Indonesia holds 16 liters. So I have to go to the store every 4 days to buy a new one or risk running out of drinkable water.

Future

I don't plan to stop tracking my water intake for now. There really is no friction logging this and the current lockdown over here makes it pretty simple to drink nothing else. I am curious to see what I learn from tracking my water intake for one year or even longer.

I drank one liter of water while writing this entry.

Current Usage

The "Screen Time" feature on my iPhone is really helpful to figure out how I currently spend my time when using it. Although, I noticed that it often does not record the times correctly. For example calls via Telegram get recorded as active usage and calls via FaceTime or cellular network do not get recorded at all. Luckily this data can be combined with the activity data from the battery settings that also shows how long an app was used in background and foreground.

My current usage unfortunately does not mirror my desired behavior. It shows that I still spend way too much time in social networks I essentially despise. For the most hours I use it to talk to important people that live on other sides of the planet. The rest of the usage seems fine to me. I chat a lot with people I can't spend time with during COVID-19, which can get distracting when promptly doing it after receiving notifications, which I usually don't. Additionally it looks like I am checking the news too often during the day. There is not really much happening that I have to keep up to date on.

Optimal Usage

The optimal phone usage would be one where I never get distracted by shallowness and only use it for things that are essential for my daily routine or connecting me with other people on a deeper level. Besides that I use utilities that make use of the phones hardware to make life easier.

Shallowness arises when I use social media apps. They are merely able to bring cheap dopamine rushes and interrupting distractions. Time spent in them does rarely feel valuable and usually leaves me with a worse feeling than before. Additionally they rob my time and focus, by sending pointless and interrupting push notifications. They make profit by stealing time and therefore are the enemy.

Essential I consider apps that I need everyday. For example an app to workout, an app to meditate and the Futureland app that helps me focus on what matters to me.

Connecting with other people is meant in a sense where real connections can evolve. This does not happen through likes or comments on various social media platforms. This category includes messengers and various voice call apps that enable conversations on deeper levels. However, I always prefer face to face communication when possible. There is and never will be a substitution for real human contact.

Utilities are apps that are not distracting in any way but make life easier. This can be everything "boring" like a calculator, calendar, camera or an app that helps me to find destinations when driving a car or scooter. These are apps that naturally only get used when they are needed. They always help but never distract.

How to Get There

The iPhone automatically sends out a notification once a week to compare the phone usage to the preceding week. It shows stuff like "Your usage has gone up by 49%". I don't think this is very useful. Not all the time spent on a phone is bad. There is nothing wrong with a long video call or a good book reading session, even though I prefer to do that on my iPad. To get more conscious and mindful about the phone usage this metric seems pretty useless.

Tools like "Screen Time" that enable me to set daily time limits for different apps seem useful at first. I gave it a try, but a brain drenched in dopamine has no difficulty to just click the "ignore limit for today" button. It is way too easy to go past the limitations. Using them alone is not enough to prevent me from using apps that provide me with a cheap thrill. Although I have to say it can help at first. My Twitter usage almost vanished over night since I am using this. The deeper problem is that one app easily gets replaced by another similar addicting one.

Push Notifications are the most toxic thing about phones. In some cases it is not even enough to go into do not disturb mode because they can get distracting even when they only show up in the notification center. I can't understand humans anymore, that let themselves get distracted by the vibrating and vibrant noises their phones make when receiving a new push notification. I wonder how they get anything done. Even worse, their mind has to be in a constant state of interruption and confusion. It is really hard to think clearly in this state of mind. Having push notifications turned on gives companies the chance to rob the users attention at any moment in the day and exploiting it for their own profit. This is why every app that sends advertisements or shallow and therefore needless notifications, immediately gets its push notification permission revoked. I can imagine that in the future we all will realize that they were a bad idea from the beginning. Similar to the "Like" button they are creating this artificial obligation that quickly leads to addiction. Our bored brains are craving for "The New" and get a big dopamine rush every time we notice a vibration or one of the playful sounds. The reason is the number one ingredient in slog machines: intermittent variable rewards.

I think that being more mindful about the problem can help. I noticed that documenting behavior and patterns can help to achieve this in other parts of my life. Screen Time can be used at the end of every day to judge if a goal was achieved or if the behavior changes for the better. My new plan is to start documenting my progress to be more clear about how I use my phone and where I am solely wasting time.

Two weeks ago I deployed my own HTTP Live Streaming (HLS) solution on Futureland. This experiment went fairly well. Don't get irritated by the name, especially the "Live Streaming" part. It does not mean that HLS exclusively can be used for live streaming purposes. As Wikipedia explains:

HTTP Live Streaming (also known as HLS) is an HTTP-based adaptive bitrate streaming communications protocol developed by Apple Inc. and released in 2009.

The most interesting part about this protocol is the "adaptive bitrate streaming" attribute. It is designed to adjust the quality of the media stream according to the user's bandwidth, CPU capacity and screen resolution. So no matter the device or location of a user, only the optimal version for the current situation gets downloaded. This results in almost no buffering, faster start times and better experience for both fast and slow internet connections.

Architecture

To provide a seamless HLS experience to our users we need the following components:

Encoder

At first all the incoming high bit rate videos have to get processed to get segmented into smaller multi-second video parts (Futureland uses 10 seconds for each segment). This can for example happen by using the awesome FFMPEG. It will generate small video chunks of equal length for every bit rate and resolution in the already pretty magic H.264 format. Additional every chunk will be encapsulated by a MPEG-2 Transport stream and kept as .ts files. After that a .m3u8 manifest file gets generated to reference all fragmented files and their respective bite rates and resolutions.

In the case of Futureland we automatically create 360p, 480p, 720p and 1080p versions of every video that gets uploaded.

Server

After the encoder is done we already can start serving the newly generated files to clients. Basically we just need a HTTP server that hosts all created .m3u8 and .ts files. All of these files can be cached as they will not be mutated anymore.

Client

After all the files are served, clients can start requesting the .m3u8 manifest. The client uses this file to determine which version of the chunks to load. It usually starts by loading the stream with the lowest bit rates. If it finds that the network is fast enough to download a segment with a higher bitrate the next chunk will be requested with a higher bit rate. In case the clients notices a lower network throughput as before it will start requesting segments with a lower bit rate again. At the same time it will automatically assemble the sequence to allow uninterrupted playback for the user.

When to use it

HLS can be used for many different purposes. It is especially useful for building:

• A platform that supports user uploaded videos like Futureland or Twitter
• A video on demand service like Youtube or Netflix
• Video into a custom CMS

HLS is highly compatible and can be used in every modern browser with the help of the Media Source Extensions API. I would advise you to use hls.js to easily embed HLS videos on your site. On Futureland we additionally provide a high quality mp4 fallback in case a the Media Source Extension API is not available for some reason. This also has the advantage that we can offer to download the videos.

Conclusion

HLS is currently the best way to provide your users with adaptive bitrate streaming and give them a responsive experience no matter their internet connection or device. The only downside I can see right now is that encoding all videos into smaller .ts segments can be expensive, takes a lot of time and of course uses more disk space than a single .mp4 video.

In case you want to see how it works and feels in practice you can check out Futureland. HLS is enabled for all video outputs since 18th of May and will be rolled out to already uploaded videos soon.

One approach that I like is to let them report to Sentry. Fortunately the Nuxt.js community has our back and already provides us with a useful Nuxt.js module.

As you can see in the readme the setup is pretty easy although I find, that the documentation is lacking some information on how to use source mapping and release management. I will cover this in this tutorial.

Setup

You can start by running npm install @nuxtjs/sentry to install the required dependency.

In the next step you have to add the module in your nuxt.config.js like this:

// nuxt.config.js
{
  modules: [
    '@nuxtjs/sentry',
  ],
  sentry: {
    dsn: process.env.SENTRY_DSN,
    publishRelease: true,
    sourceMapStyle: 'hidden-source-map',
    config: {
      release: process.env.GIT_COMMIT_SHA,
    },
  },
  // ***
}

As you can see we will use two environment variables to configure the integration. As you probably already guessed should you provide the project DSN via SENTRY_DSN.

What you use in the release option is up to you. You can use the version string of your package.json or whatever you want. In my opinion it makes sense to use something like the Git commit SHA, so you exactly know in which commit this error occurred. In most continues integration tools you will be provided with environment variables give you the needed information to do this.

What took me quite some time to figure out and is the reason why I wrote this little tutorial is that you have to include a .sentryclirc file in your root directory of your project.

It's contents should look something like this:

# .sentryclirc

[defaults]
org = <your_sentry_org>
project = <your_sentry_project>
url = <your_sentry_url>

When you are done with all this you are good to go. The module will upload the source maps to Sentry and create new releases at build-time. This behavior is especially helpful when using Nuxt.js because all the Javascript assets usually get minified which means that debugging is almost impossible and stack traces would otherwise be totally unreadable.

What do we need to generate pretty link previews? The same data everyone else needs. That is why we expose some robot friendly meta data on all of our Futureland pages. This information includes data about your profiles, projects and outputs. We expose this data by using the Open Graph Protocol. Every modern website uses this protocol to provide third party services with data about their service. This data is used to generate pretty previews of your links in Slack, Telegram, Twitter and so on.

The format of that meta data are standardized HTML tags and should appear in the <head> section of an website.

On Futureland these tags look something like this:

<head>
    ...
    <meta property="og:title" content="lucas - Futureland" />
    <meta
        property="og:description"
        content="Making some small improvements to profiles. Adjusting spacing of key information and adding a space for a url."
    />
    <meta
        property="og:image"
        content="https://cdn.futureland.tv/app/internetvin/549/thumbnail_A5xKReffB5dLcY0fSUtGJ.png"
    />
    <meta property="og:image:width" content="3584" />
    <meta property="og:image:height" content="2278" />
    ...
</head>

As you can see we give those services some basic information about our page. This is enough to render a nice link preview including an image thumbnail.

For Futureland to be able to create beautiful link previews we first have to make a request to the URL the preview should be generated for.

After we do that we can query all of the Open Graph Data if it exists. Then we save the corresponding data and files for later. That means we download every image and video to our server and save the rest of the information in our database.

Since we now have all the required data to generate a fancy link preview we are done.

Node.js

There is a handy Node.js library called open-graph-scraper that you can use to obtain Open Graph Data easily.

const ogs = require('open-graph-scraper')

const { data } = await ogs({ url: 'https://futureland.tv' })

/*
{
    ogImage: 'https://cdn.futureland.tv/Images/project%20_grid.jpg',
    ogTitle: 'Futureland',
    ogDescription:
        'A time machine, feedback engine, and fund for your projects.',
    ogUrl: 'https://futureland.tv',
}
*/

The goal is to have a single Docker image for every service in the repository. That means every service should have its own Dockerfile to make the image as slim as possible.

Before writing this I tried to find help for this problem myself. The most tutorials recommended using something like Lerna or Yarn workspaces. Both of this solutions felt way to bloated for this simple goal of sharing a little bit of common source code. After all the repo should stay small and clean.

Another solution that often got recommended was to create a private NPM package or a self hosted package via git that includes all of the shared source code. My goal was to keep this a monorepo so that was not really an option.

NPM and local packages

As many of you probably already know there is a simple way in npm to install local packages as dependencies. So you can type something like npm install ../shared to install a local package without hosting it on NPM or using a second git repository. While developing you can even use npm link ../shared inside of Docker to always have the latest version of your shared directory.

In your services you then can use something like const { service } = require('shared') to access the shared package.

The only thing left to do is creating a Dockerfile that is able to install the shared package. To be able to do this the first change has to be to change the Docker context to the root folder of the repository so that Docker has access to all the files it needs.

That said you only have to change the directory paths and copy in the shared folder path like COPY ./shared ./shared

The resulting Dockerfile looks something like this:

FROM mhart/alpine-node:14

WORKDIR /srv

COPY worker/package.json worker/package-lock.json ./worker/
COPY ./shared ./shared

RUN cd worker && npm ci

COPY ./worker ./worker

WORKDIR /srv/worker

CMD NODE_ENV=production node index.js

Important is that before you can built this Dockerfile you have to run npm install ../shared to add a new entry to the package.json of your service. Your package.json should have a new entry like "shared": "file:../shared" In the same step NPM will link all needed dependencies in package-lock.json. This step has to be repeated every time the dependencies of the shared package will change.

Conclusion

Using a NodeJS Docker monorepo with locally shared dependencies does not have to be hard. You don't have to use Lerna or Yarn workspaces. It is enough to use the built-in tools from NPM.

In my opinion having the shared dependency in the same repository as all the services has a few advantages:

• With a rock solid testing suite changes on the shared package can be made much faster and more confident
• Enables testing the whole application at once
• Updates in the shared package automatically apply for all services that use it
• Not dealing with extra NPM packages or git dependencies

This post will be about a new caching technique i recently learned about and use in the projects i am working on, including this very blog. The concept is called stale-while-revalidate and will result in loading times that are blazing fast all around the world.

How does it work?

When you use a CDN-proxy in front of your website all of your websites traffic will go through that proxy. Examples for this are Cloudflare, AWS Cloudfront and the smart ZEIT Smart CDN.

This proxy will try to figure out how to cache your websites traffic based on HTTP-headers. The most important header for this blog post will be Cache-Control. It usually specifies how long your browser should cache a certain asset or image. The new stale-while-revalidate directive tells the proxy to always serve a stale version of a file (if possible) and revalidate the cache in the background. This results in fast load times all around the world because the user will always get an already cached version of a file. Even if the request would take multiple seconds to resolve, with the help of the proxy the user always gets his response blazing fast.

I recommend to use the Cache-Control header with the following directives:

Cache-Control: s-maxage=1, stale-while-revalidate

The stale-while-revalidate directive tells the proxy to revalidate the cache in the background. At the same time s-maxage=1 makes sure to only revalidate once every second. This means even when there are thousands of requests hitting your domain in one second, the proxy will only make one request to your server and validate if the cache needs to be updated. That behavior makes serving your content incredibly cheap and fast at the same time.

Be aware that the cache needs time to get hot. The very first request will be served synchronously because the proxy has to create a new cached version first, after that all subsequent requests are served from cache. If the cache is stale, revalidation will happen asynchronously in the background. If the cache is stale is dependent on the s-maxage=<seconds> value. To always get the latest version i recommend to use it with a value of 1, which means one second.

Demo

At the time i am writing this post i am sitting in Bali and my server that is running the Ghost instance is back in Germany. A ping gives me a latency of ~200 ms.

After i make a request to https://lucas.love you can see that it automatically uses the nearest CDN, wich in my case is in Singapore. The request only takes up 271 ms.

Let's try the Pragma: no-cacheheader. This will tell the CDN to skip the cache and make a new synchronous request to the origin server. It will simulate the the same behavior as if it would be the very first request that hits the CDN with a cold cache.

You can see that the request takes up a lot more time, 1487 ms to be precise. Way slower than the request before. This happens because my server first has to look up some stuff in it's database and responds over a second later (Server Processing).

Fewer Downtimes

One additional positive side-effect to know is that it also helps with having fewer downtimes. If the proxy is not able to get a fresh version of your file because your origin server is currently down, it will still give all clients the latest working copy without any form of interruption.

Downsides

Because the client will always get a stale version it is possible that a user visiting your site will get a very old version that for example does not contain the latest changes you made to your blog.

To avoid this you have to build your own system around that problem. lucas.love for example tells your browser to make a XHR request with the Pragma: no-cache header to the same URL you are currently visiting after the page finished loading and when you reopen the browser tab. After the request was made the browser will compare a header called x-head that contains a hash of the contents of the page. If this header does not match, your browser will reload the page with the latest copy available. Because the proxy will use your XHR request to revalidate the cache this will happen so fast that your users probably will not even notice it.

Conclusion

Stale While Revalidate is a very powerful concept. It takes load of your server and at the same time makes your site faster for all users around the world. I would recommend to use this directive for the following use-cases:

• Blogs and Newspapers: Your content may change infrequently but you still have the flexibility to fix typos for example. You don't have to wait for the cache to expire.
• APIs: The content may change frequently but takes a significant amount of time to generate. For example low database queries and high latencies.
• Public dashboards: Lots of users consume loads of dynamic content.
• Webshops: with lots of products and categories.
• Landing page: Changes infrequently but has a lot of users that access it at the same time.

Do not try to use the stale-while-revalidate when content of your site can not be cached. For example when you have to deal with user authentication or have to render user specific content like on a social network. If you can't use a cache, stale-while-revalidate is not the right use-case for your project.