19 Aug 2019

feedPlanet Grep

Steven Wittens: Hello World on the GPU

An Easy Tutorial

Graphics programming can be intimidating. It involves a fair amount of math, some low-level code, and it's often hard to debug. Nevertheless I'd like to show you how to do a simple "Hello World" on the GPU. You will see that there is in fact nothing to be afraid of.

Most environments offer you a printf-equivalent and a string type, but that's not how we do things in GPU land. We like the raw stuff, and we work with pixels themselves. So we're going to draw our text to the console directly. I'll show you the general high level flow, and then wrap up some of the details.

a window saying hello world

First, we're going to define our alphabet.

let alphabet = allocate_values(
  &['H', 'E', 'L', 'O', 'W', 'R', 'D', '🌎'],
  AlphabetFormat::Letter);

Next we define our message by encoding it from this alphabet.

let message = allocate_indices(
//  H  E  L  L  O  W  O  R  L  D
  &[0, 1, 2, 2, 3, 4, 3, 5, 2, 6, 7],
  IndexFormat::Uint8);

We'll also need to assemble this alphabet soup into positioned text. Don't worry, I precalculated the horizontal X offsets:

let xs = allocate_values(
  &[0.0, 49.0, 130.0, 195.0, 216.0, 238.0, 328.0, 433.0, 496.0, 537.0, 561.0, 667.0],
  AlphabetFormat::Float);

The font is loaded as glyphs, a map of glyph images:

let glyphs = console.load_glyphs("helvetica.ttf");

We now have everything we need to print it pixel-by-pixel to the top of the console, which we call 'rasterizing':

fn hello_world(
  line: Line,
  message: Vec<Index>,
  alphabet: Vec<Letter>,
  xs: Vec<Float>,
  args: Vec<Argument>,
) {
  // Get glyph library
  let glyphs = args[0];

  // Loop over all the indices in the message
  for i in 0..message.len() {

    // Retrieve the x position for this index.
    let x = xs[i];

    // Retrieve the letter in the alphabet
    let letter = alphabet[message[i]];
    
    // Retrieve the glyph image for this letter
    let glyph  = glyphs[letter];

    // Rasterize it to the line
    rasterize(line, x, glyph.image, glyph.width, glyph.height);
  }
}

rasterize() is provided for you, but if you're curious, this is what it looks like on the inside:

fn rasterize(
  line: Line,
  offset: Float,
  image: Frame,
  width: Int,
  height: Int
) {

  // Iterate over rows and columns
  for y in 0..height {
    for x in 0..width {

      // Get target position
      let tx = x + offset;
      let ty = y;

      // Get image pixel color
      let source = get(image, x, y);

      // Get current target pixel color
      let destination = get(line, tx, ty);

      // Blend source color with destination color
      let blended = blend(source, destination);

      // Save new color to target
      set(target, tx, ty, blended);
    }
  }
};

It's just like blending pixels in Photoshop, with a simple nested rows-and-columns loop.

Okay so I did gloss over an important detail.

The thing is, you can't just call hello_world(...) to run your code. I know it looks like a regular function, just like rasterize(), but it turns out you can only call built-in functions directly. If you want to call one of your own functions, you need to do a little bit extra, because the calling convention is slightly different. I'll just go over the required steps so you can follow along.

First you need to actually access the console you want to print to.

So you create a console instance:

let instance = Console::Instance::new();

and get an adapter from it:

let adapter =
  instance.get_adapter(
    &AdapterDescriptor {
      font_preference: FontPreference::Smooth,
    });

so you can get an actual console:

let console =
  adapter.create_console(
    &ConsoleDescriptor {
      extensions: Extensions {
        subpixel_antialiasing: true,
      },
    });

But this console doesn't actually do anything yet. You need to create an interactive window to put it in:

let events_loop = EventsLoop::new();

let window = WindowBuilder::new()
  .with_dimensions(LogicalSize {
    width: 1280.0, height: 720.0
  })
  .with_title("Console")
  .build(&events_loop).unwrap();

and then make a surface to draw to:

let surface = instance.create_surface();

Now if you want to print more than one line of text, you need to set up a line feed:

let descriptor =
  LineFeedDescriptor {
    usage: LineUsageFlags::OUTPUT_ATTACHMENT,
    format: TextFormat::UTF8,
    width: 120,
    height: 50,
  };

let line_feed = console.create_line_feed(&surface, &descriptor);

let next_line = line_feed.get_next_line();

And if you want emoji, which we do, you need a separate emoji buffer too:

let images =
  console.create_emoji(
    EmojiDescriptor {
      size: Extent2d {
        width: 256,
        height: 256,
      },
      array_size: 1024,
      dimension: ImageDimension::D2,
      format: ImageFormat::RGBA8,
      usage: ImageUsageFlags::OUTPUT_ATTACHMENT,
    });

let emoji_buffer = images.create_default_view();

Okay, we're all set!

Now we just need to encode the call, using a call encoder:

let encoder = console.create_call_encoder();

We begin by describing the special first argument (line), a combo of next_line and the emoji_buffer. We also have to provide some additional flags and parameters:

let call =
  encoder.encode_function_call(
    FunctionCallDescriptor {
      console_attachments: &[
        ConsoleAttachmentDescriptor {
          attachment: &next_line,
          load_op: LoadOp::Clear,
          store_op: StoreOp::Store,
          clear_letter: ' ',
        }
      ],
      emoji_attachment: Some(
        ConsoleEmojiAttachmentDescriptor {
          attachment: &emoji_buffer,
          load_op: LoadOp::Clear,
          store_op: StoreOp::Store,
          clear_color: "rgba(0, 0, 0, 0)",
        })
    });

The message of type Vec<Index> is added using a built-in convention for indices:

call.set_index_buffer(message);

The alphabet: Vec<Letter> and the xs: Vec<Float> can also be directly passed in, because they are accessed 1-to-1 using our indices, as numbered arguments:

call.set_alphabet_buffers(&[
  (&alphabet, 0), (&xs, 1)
]);

However, the glyph images are a bit trickier, as they are a custom keyword argument.

To make this work, we need to create an argument group layout, which describes how we'll pass the arguments to sample our glyph images:

let argument_group_layout =
  console.create_argument_group_layout(
    &ArgumentGroupLayoutDescriptor {
      bindings: &[
        ArgumentGroupLayoutBinding {
            binding: 0,
            visibility: Visibility::PIXEL,
            ty: BindingType::SampledText,
        },
        ArgumentGroupLayoutBinding {
            binding: 1,
            visibility: Visibility::PIXEL,
            ty: BindingType::Sampler,
        },
      ]
    });

We then put it into a larger function call layout, in case we have multiple groups of keyword arguments:

let function_call_layout =
  console.create_function_call_layout(
    FunctionCallLayoutDescriptor {
      argument_group_layouts: &[argument_group_layout],
    });

We also need to create bindings to match this layout, to actually bind our argument values:

let glyph_view = glyphs.create_default_view();

let sampler = console.create_sampler(
  &TextSamplerDescriptor {
    address_mode: AddressMode::ClampToEdge,
    text_filter: FilterMode::TypeHinted,
    hint_clamp: 100.0,
    max_anisotropy: 4,
    compare_function: CompareFunction::Always,
    border_color: BorderColor::TransparentBlack,
  });

let argument_group =
  console.create_argument_group(
    &BindGroupDescriptor {
      layout: argument_group_layout,
      bindings: &[
        Binding {
          binding: 0,
          resource: BindingResource::ImageView(&glyph_view),
        },
        Binding {
          binding: 1,
          resource: BindingResource::Sampler(&sampler),
        },
      ]
    });

And add it to our call:

call.set_argument_group(0, argument_group);

Alright! We're pretty much ready to make the call now. Just one more thing. The function call descriptor.

We need to pass the raw code for hello_world as a string to console.create_code_module, and annotate it with a few extra bits of information:

let function_call =
  console.create_function_call(
    &FunctionCallDescriptor {
      layout: &function_call_layout,
      call_stage: CallStageDescriptor {
        module: console.create_code_module(&hello_world),
        entry_point: "hello_world",
      },
      rasterization_state: RasterizationStateDescriptor {
        emoji_alignment: Alignment::Middle,
        emoji_bias: 0,
        emoji_scale: 1.5,
      },
      text_topology: Topology::Letters,
      console_states: &[
        ConsoleStateDescriptor {
          format: TextFormat::UTF8,
          color: BlendDescriptor {
            src_factor: BlendFactor::SrcAlpha,
            dst_factor: BlendFactor::OneMinusSrcAlpha,
            operation: BlendOperation::Add,
          },
          alpha: BlendDescriptor {
            src_factor: BlendFactor::OneMinusDstAlpha,
            dst_factor: BlendFactor::One,
            operation: BlendOperation::Add,
          },
          write_mask: ColorWriteFlags::ALL,
        },
      ],
      emoji_state: Some(EmojiStateDescriptor {
        format: ImageFormat::RGBA8,
        emoji_enabled: true,
        emoji_variant: CompareFunction::LessEqual,
      }),
      index_format: IndexFormat::Uint8,
      alphabet_buffers: &[
        AlphabetBufferDescriptor {
          stride: 1,
          step_mode: InputStepMode::Letter,
          attributes: AlphabetAttributeDescriptor {
            attribute_index: 0,
            format: AlphabetFormat::Letter,
            offset: 0,
          },
        },
        AlphabetBufferDescriptor {
          stride: 1,
          step_mode: InputStepMode::Letter,
          attributes: AlphabetAttributeDescriptor {
            attribute_index: 1,
            format: AlphabetFormat::Number,
            offset: 0,
          },
        },
      ],
      sample_count: 1,
    });

Which we add to the call:

call.set_function_call(&function_call);

Well, you actually have to do this first, but it was easier to explain it last.

Now all that's left is to submit the encoded command to the console queue, and we're already done:

console
  .get_queue()
  .submit(&[encoder.finish()]);

a black window

Hm.

Damn, and I was going to show you how to make a matrix letter effect as an encore. You can pass a letter_shader to rasterizeWithLetterFX(...). It's easy, takes a couple hundred lines tops, all you have to do is call a function on a GPU.

(All code in this post is real, but certain names and places have been changed to protect the innocent. If you'd like to avoid tedious bureaucracy in your code, why not read about how the web people are trying to tame similar lions?)

12
Objects created

19 Aug 2019 10:00pm GMT

Dries Buytaert: Low-code and no-code tools continue to drive the web forward

Low code no code

A version of this article was originally published on Devops.com.

Twelve years ago, I wrote a post called Drupal and Eliminating Middlemen. For years, it was one of the most-read pieces on my blog. Later, I followed that up with a blog post called The Assembled Web, which remains one of the most read posts to date.

The point of both blog posts was the same: I believed that the web would move toward a model where non-technical users could assemble their own sites with little to no coding experience of their own.

This idea isn't new; no-code and low-code tools on the web have been on a 25-year long rise, starting with the first web content management systems in the early 1990s. Since then no-code and low-code solutions have had an increasing impact on the web. Examples include:

While this has been a long-run trend, I believe we're only at the beginning.

Trends driving the low-code and no-code movements

According to Forrester Wave: Low-Code Development Platforms for AD&D Professionals, Q1 2019, In our survey of global developers, 23% reported using low-code platforms in 2018, and another 22% planned to do so within a year..

Major market forces driving this trend include a talent shortage among developers, with an estimated one million computer programming jobs expected to remain unfilled by 2020 in the United States alone.

What is more, the developers who are employed are often overloaded with work and struggle with how to prioritize it all. Some of this burden could be removed by low-code and no-code tools.

In addition, the fact that technology has permeated every aspect of our lives - from our smartphones to our smart homes - has driven a desire for more people to become creators. As the founder of Product Hunt Ryan Hoover said in a blog post: As creating things on the internet becomes more accessible, more people will become makers..

But this does not only apply to individuals. Consider this: the typical large organization has to build and maintain hundreds of websites. They need to build, launch and customize these sites in days or weeks, not months. Today and in the future, marketers can embrace no-code and low-code tools to rapidly develop websites.

Abstraction drives innovation

As discussed in my middleman blog post, developers won't go away. Just as the role of the original webmaster (FTP hand-written HTML files, anyone?) has evolved with the advent of web content management systems, the role of web developers is changing with the rise of low-code and no-code tools.

Successful no-code approaches abstract away complexity for web development. This enables less technical people to do things that previously could only be done by developers. And when those abstractions happen, developers often move on to the next area of innovation.

When everyone is a builder, more good things will happen on the web. I was excited about this trend more than 12 years ago, and remain excited today. I'm eager to see the progress no-code and low-code solutions will bring to the web in the next decade.

19 Aug 2019 8:35am GMT

14 Aug 2019

feedPlanet Grep

Philip Van Hoof: In Varietate Concordia

Al vraag ik me af waar de permanente Vlaamse vertegenwoordiging voor de Europese Commissie en Unie zich mee bezig houdt en waarom ze zo weinig van zich laten horen? Dat lijstje prioritaire dossiers is namelijk maar erg mager. Er waren bv. toch handelsovereenkomsten in 2019? Is onze permanente vertegenwoordiging daar dan niet mee bezig geweest? Ik vind er weinig over terug.

Zijn er voor 2020 dan geen handelsovereenkomsten in de maak? Ik dacht het wel. Daar is Vlaanderen niet mee bezig? Dat vindt onze permanente vertegenwoordiging het niet waard om te vermelden op hun website? En zo ja, waar dan?

Zoals ik het begrepen heb spreken al onze diplomaten in Brussel Frans en, veel belangrijker dan dat (want mij part spreken ze het Afrikaans), vertolken ze het Belgische (en eigenlijk ook het Franse) standpunt. Zelden dat van Vlaanderen en nog minder krijgt dit de nodige politieke -en media aandacht.

De NV-A wil vaak doen alsof ze het niveau van de Europese Unie belangrijk vinden. Maar eigenlijk bieden ze maar weinig aan, doen ze er maar weinig mee. En daardoor zijn we als Vlaanderen helemaal niet of maar mager vertegenwoordigd.

Ik had graag in Bart De Wever's nota wat meer duidelijk gehad in plaats van enkel een Latijnse spreuk. Spreuken en getwitter is geen beleid voeren, Bart. Het wordt tijd dat uw partij i.p.v. enkel grote uitspraken uit te kramen en het populisme aan te wakkeren ook eens toont dat het kan besturen.

Ja ja. We gaan onafhankelijk worden in Europa en de Europese Unie. Dat is duidelijk. Maar hoe gaat de NV-A onze permanente vertegenwoordiging van Vlaanderen voor de EU dan versterken?

Een onafhankelijk Vlaanderen moet niet verwachten dat Belgische diplomaten Vlaamse kastanjes uit het vuur gaan halen.

Wees concreet.

14 Aug 2019 9:44am GMT

13 Aug 2019

feedPlanet Grep

Mattias Geniar: Time for Change: Going Independent

The post Time for Change: Going Independent appeared first on ma.ttias.be.

After 12 intense years at Nucleus, it's time for something new: as of September 2019 I'll stop my activities at Nucleus and continue to work as an independent, focussing on Oh Dear!, DNS Spy & Syscast.

The road to change

Why change? Why give up a steady income, health- & hospital insurance, a company car, paid holidays, fun colleagues, exciting tech challenges, ... ?

I think it's best explained by showing what an average day looked like in 2016-2017, at the peak of building DNS Spy.

Back when I had the idea to create a DNS monitoring service, the only way I could make it work was to code on it at crazy hours. Before the kids woke up and after they went to bed. Before and after the more-than-full-time-job.

This worked for a surprisingly long time, but eventually I had to drop the morning hours and get some more sleep in.

Because of my responsibilities at Nucleus (for context: a 24/7 managed hosting provider), I was often woken during the night for troubleshooting/interventions. This, on top of the early hours, made it impossible to keep up.

After a while, the new rhythm became similar, but without the morning routine.

Notice anything missing in that schedule? Household chores? Some quality family time? Some personal me-time to relax? Yeah, that wasn't really there.

There comes a point where you have to make a choice: either continue on this path and end up wealthy (probably) but without a family, or choose to prioritize the family first.

As of September 2019, I'll focus on a whole new time schedule instead.

A radical (at least for me) change of plans, where less time is spent working, more time is spent with the kids, my wife, the cats, the garden, ...

I'm even introducing a bit of whatever-the-fuck-i-want-time in there!

What I'll be working on

In a way I'm lucky.

I'm lucky that I spent the previous 10+ years working like a madman, building profitable side businesses and making a name for myself in both the open source/linux and PHP development world. It allows me to enter September 2019 without a job, but with a reasonable assurance that I'll make enough money to support my family.

How?

For starters, I'll have more time & energy to further build on DNS Spy & Oh Dear!. These 2 side businesses will from now on be called "businesses", as they'll be my main source of income. It isn't enough to live on, mind you, so there's work to be done. But at least there's something there to build on.

Next to that, my current plan is to revive and start building on Syscast. The idea formed in 2016 (the "workaholic" phase, pre-DNS Spy) and was actually pretty fleshed out already. Making online courses, building upon the 10+ years of sysadmin & developer knowledge.

Syscast didn't happen in 2016 and pivoted to a podcast that featured impressive names like Daniel Stenberg (curl & libcurl), Seth Vargo (Hashicorp Vault), Matt Holt (Caddy) and many others instead.

I've always enjoyed giving presentations, explaining complicated technologies in easy terms and guiding people to learn new things. Syscast fits that bill and would make for a logical project to work on.

Looking back at an amazing time

A change like this isn't taken lightly. Believe me when I say I've been debating this for some time.

I'm grateful to both founders of Nucleus, Wouter & David, that they've given me a chance in 2007. I dropped out of college, no degree, just a positive attitude and some rookie PHP knowledge. I stumbled upon the job by accident, just googling for a PHP job. Back then, there weren't that many. It was either Nucleus or a career writing PHP for a bank. I think this is where I got lucky.

I've learned to write PHP, manage Linux & Windows servers, do customer support, how to do marketing, keep basic accounting and the value of underpromise and overdeliver. I'll be forever grateful to both of them for the opportunity and the lessons learned.

It was also an opportunity to work with my best friend, Jan, for the last 9 years. Next to existing friends, I'm proud to call many of my colleagues friends too and I hope we can stay in touch over the years. I find relationships form especially tight in intense jobs, when you heavily rely on each other to get the job done.

Open to new challenges

In true LinkedIn parlance, I'm open to new challenges. That might be a couple of days of consultancy on Linux, software architecture, PHP troubleshooting, scalability advice, a Varnish training, ...

I'm not looking for a full-time role anywhere (see the time tables above), but if there's an interesting challenge to work on, I'll definitely consider it. After all, there are mouths to feed at home. ;-)

If you want to chat, have a coffee, exchange ideas, brainstorm or revolutionize the next set of electric cars, feel free to reach out (my contact page has all the details).

But first, a break

However, before I can start doing any of that, I need a time-out.

In September, my kids will go to school and things will be a bit more quiet around the house. After living in a 24/7 work-phase for the last 10 years, I need to cool down first. Maybe I'll work on the businesses, maybe I won't. I have no idea how hard that hammer will hit come September when I suddenly have my hands free.

Maybe I'll even do something entirely different. Either way, I'll have more time to think about it.

The post Time for Change: Going Independent appeared first on ma.ttias.be.

13 Aug 2019 7:45am GMT

12 Aug 2019

feedPlanet Grep

Dries Buytaert: Increasing Drupal speaker diversity

A special bird flying in space has the spotlight while lots of identical birds sit on the ground (lack of diversity)

At Drupalcon Seattle, I spoke about some of the challenges Open Source communities like Drupal often have with increasing contributor diversity. We want our contributor base to look like everyone in the world who uses Drupal's technology on the internet, and unfortunately, that is not quite the reality today.

One way to step up is to help more people from underrepresented groups speak at Drupal conferences and workshops. Seeing and hearing from a more diverse group of people can inspire new contributors from all races, ethnicities, gender identities, geographies, religious groups, and more.

To help with this effort, the Drupal Diversity and Inclusion group is hosting a speaker diversity training workshop on September 21 and 28 with Jill Binder, whose expertise has also driven major speaker diversity improvements within the WordPress community.

I'd encourage you to either sign up for this session yourself or send the information to someone in a marginalized group who has knowledge to share, but may be hesitant to speak up. Helping someone see that their expertise is valuable is the kind of support we need to drive meaningful change.

12 Aug 2019 7:44pm GMT

FOSDEM organizers: Call for participation

We now invite proposals for main track presentations, developer rooms, stands and lightning talks. FOSDEM offers open source and free software developers a place to meet, share ideas and collaborate. Renowned for being highly developer-oriented, the event brings together some 8000+ geeks from all over the world. The twentieth edition will take place on Saturday 1st and Sunday 2nd February 2020 at the usual location: ULB Campus Solbosch in Brussels. We will record and stream all main tracks, devrooms and lightning talks live. The recordings will be published under the same licence as all FOSDEM content (CC-BY). If, exceptionally,舰

12 Aug 2019 3:00pm GMT

10 Aug 2019

feedPlanet Grep

FOSDEM organizers: Next FOSDEM: 1 & 2 February 2020

FOSDEM 2020 will take place at ULB Campus Solbosch on Saturday 1 and Sunday 2 February 2020. Further details and calls for participation will be announced in the coming days and weeks.

10 Aug 2019 3:00pm GMT

09 Aug 2019

feedPlanet Grep

Steven Wittens: Model-View-Catharsis

MVC was a mistake

A reader sent in the following letter:

Dear Mr. Wittens,

I have read with great interest your recent publications in Computers Illustrated. However, while I managed to follow your general exposition, I had trouble with a few of the details.

Specifically, Googling the phrase "topping from the bottom" revealed results that I can only describe as being of an unchristian nature, and I am unable to further study the specific engineering practice you indicated. I am at a loss in how to apply this principle in my own professional pursuits, where I am currently tasked with building web forms for a REST API using React. Should I have a conversation about this with my wife?

With respect,
Gilbert C.

Dear Gilbert,

In any marriage, successful copulation requires commitment and a mutual respect of boundaries. Unlike spouses however, components are generally not monogamous, and indeed, best results are obtained when they are allowed to mate freely and constantly, even with themselves. They are quite flexible.

The specifics of this style are less important than the general practice, which is about the entity ostensibly in control not actually being in control. This is something many form libraries get wrong, React or not, missing the true holy trinity. I will attempt to explain. This may take a while. Grab some hot chocolate.

Simon Stålenhag Painting
Simon Stålenhag

Infinite omnipotent hyperbeing lovingly screaming "YOU'RE VALID", its song echoed across every structural detail of the universe-you, an insignificant mote adrift in some galactic tide shouting back "YES, OKAY, WHAT ELSE?"

Fragnemt by ctrlcreep
(a linguistic anti-depressant in book form)

That's Numberwang

In your web forms, you're building a collection of fields, and linking them to a data model. Either something to be displayed, or something to be posted, possibly both. Or even if it's a single-page app and it's all for local consumption only, doesn't really matter.

Well, it does, but that's boundary number 1. A form field shouldn't know or care about the logistics of where its data lives or what it's for. Its only task is to specialize the specific job of instantiating the value and manipulating it.

There's another boundary that we're also all used to: widget type. A text field is not a checkbox is not a multi-select. That's boundary number 3. It's the front of the VCR, the stuff that's there so that we can paw at it.

Would you like to buy a vowel for door number 2?

P_L_C_: What APIs are about.
"Previously, on Acko."

You see, when God divided the universe into controlled (value = x) and uncontrolled (initialValue = defaultValue = x) components, He was Mistaken. A true uncontrolled component would have zero inputs.

What you actually have is a component spooning with an invisible policy. Well, if you can't pull them apart, I guess they're not just spooning. But these two are:

<ValidatingInput
  parse={parseNumber} format={formatNumber}
  value={number}    setValue={setNumber}
>{
  (value, isError, onChange, onBlur) =>
    <TextField
      value={value}      isError={isError}
      onChange={onChange} onBlur={onBlur} />
}</ValidatingInput>

This ValidatingInput ensures that a value can be edited in text form, using your parser and serializer. It provides Commitment, not Control. The C in MVC was misread in the prophecies I'm afraid. Well actually, someone spilled soda on the scrolls, and the result was a series of schisms over what they actually said. Including one ill-fated attempt to replace the Controller with a crazy person.

Point is, as long as the contents of the textfield are parseable, the value is plumbed through to the outside. When it's not, only the text inside the field changes, the inner value. The outside value doesn't, remaining safe from NaNful temptations. But the TextField remains in control, only moderated by the two policies you passed in. The user is free to type in unparseable nonsense while attempting to produce a valid value, rather than having their input messed around with as they type. How does it know it's invalid? If the parser throws an Error. A rare case of exceptions being a good fit, even if it's the one trick VM developers hate.

An "uncontrolled component" as React calls it has a policy to never write back any changes, requiring you to fish them out manually. Its setValue={} prop is bolted shut.

Note our perspective as third party: while we cannot see or touch ValidatingInput's inner workings, it still lets us have complete control over how it actually gets used, both inside and out. You wire it up to an actual widget yourself, and direct its behavior completely. It's definitely not a controller.

This is also why, if you pass in JSON.parse and JSON.stringify as the parser/formatter, with the pretty printer turned on, you get a basic but functional JSON editor for free.

You can also have a DirectInput, to edit text directly in its raw form, without any validation or parsing. Because you wrote that one first, to handle the truly trivial case. But then later you realized it was just ValidatingInput with identity functions glued into the parse/format props.

So let's talk about MVC and see how much is left once you realize the above looks suspiciously like a better, simpler recipe for orthogonalized UI.

Wikipedia - Model-View-Controller

Look at this. It's from Wikipedia so it must be true. No really, look at it closely. There is a View, which only knows how to tell you something. There is a Controller, which only knows how to manipulate something.

This is a crack team of a quadriplegic and a blind man, each in separate rooms, and you're delegating your UI to them? How, in the name of all that is holy, are you supposed to do that? Prisoner dilemmas and trolleys? No really, if the View is a TextField, with a keyboard cursor, and mouse selection, and scrolling, and Japanese, how is it possible to edit the model backing it unless you are privy to all that same information? What if it's a map or a spreadsheet? ばか!

This implies either one of two things. First, that the Controller has to be specialized for the target widget. It's a TextController, with its own house key to the TextView, not pictured. Or second, that the Controller doesn't do anything useful at all, it's the View that's doing all the heavy lifting, the Controller is just a fancy setter. The TextModel certainly isn't supposed to contain anything View-specific, or it wouldn't be MVC.

Wikipedia does helpfully tell us that "Particular MVC architectures can vary significantly from the traditional description here." but I would argue the description is complete and utter nonsense. Image-googling MVC is however a fun exercise for all the zany diagrams you get back. It's a puzzle to figure out if any two actually describe the same thing.

Model-View-Controller 1 Model-View-Controller 6
Model-View-Controller 2 Model-View-Controller 5
Model-View-Controller 4 Model-View-Controller 3

Nobody can agree on who talks to who, in what order, and who's in charge. For some, the Controller is the one who oversees the entire affair and talks to the user, maybe even creates the Model and View. For others, it's the View that talks to the user, possibly creating a Controller and fetching a Model. For some, the Model is just a data structure, mutable or immutable. For others it's an interactive service to both Controller and View. There may be an implicit 4th observer or dispatcher for each trio. What everyone does agree on, at least, is that the program does things, and those things go in boxes that have either an M, a V or a C stamped on them.

One dimension almost never illustrated, but the most important one, is the scope of the MVC in question.

Widget-MVC

Sometimes MVC is applied at the individual widget level, indeed creating a TextModel and a TextView and a TextController, which sits inside a FormModel/View/Controller, nested inside an ActivityModel/View/Controller. This widget-MVC approach leads to a massively complected tree, where everything comes in triplicate.

The models each contain partial fragments of an original source repeated. Changes are sometimes synced back manually, easily leading to bugs. Sometimes they are bound for you, but must still travel from a deeply nested child all the way back through its parents. If a View can't simply modify its Model without explicitly tripping a bunch of other unrelated Model/Controller/View combos, to and fro, this is probably an architectural mistake. It's a built-in game of telephone played by robots.

I didn't try to draw all this, because then I'd also have to define which of the three has custody of the children, which is a very good question.

Rails-MVC

But sometimes people mean the exact opposite, like client/server MVC. There the Model is a whole database, the Controller is your web application server and the View is a templated client-side app. Maybe your REST API is made up of orthogonal Controllers, and your Models are just the rows in the tables pushed through an ORM, with bells on and the naughty bits censored. While this explanation ostensibly fits, it's not accurate. A basic requirement for MVC is that the View always represents the current state of the Model, even when there are multiple parallel Views of the same thing. REST APIs simply don't do that, their views are not live. Sorry Rails.

It is in fact this last job that is difficult to do right, and which is the problem we should not lose sight of. As ValidatingInput shows, there is definitely a requirement to have some duplication of models (text value vs normalized value). Widgets definitely need to retain some local state and handlers to manage themselves (a controller I guess). But it was never about the boxes, it was always about the arrows. How can we fix the arrows?

(An audience member built like a greek statue stands up and shouts: "Kleisli!" A klaxon sounds and Stephen Fry explains why that's not the right answer.)

The Shape of Things To Come

In my experience, the key to making MVC work well is to focus on one thing only: decoupling the shape of the UI from the shape of the data. I don't just mean templating. Because if there's one thing that's clear in modern apps, it's that widgets want to be free. They can live in overlays, side drawers, pop-overs and accordions. There is a clear trend towards canvas-like apps, most commonly seen with maps, where the main artifact fills the entire screen. The controls are entirely contextual and overlaid, blocking only minimal space. Widgets also move and transition between containers, like the search field in the corner, which jumps between the map and the collapsible sidebar. The design is and should be user-first. Whether data is local or remote, or belongs to this or that object, is rarely a meaningful distinction for how you display and interact with it.

Google Maps

So you don't want to structure your data like your UI, that would be terrible. It would mean every time you want to move a widget from one panel to another, you have to change your entire data model end-to-end. Btw, if an engineer is unusually resistant to a proposed UI change, this may be why. Oops. But what shape should it have then?

You should structure your data so that GETs/PUTs that follow alike rules can be realized with the same machinery through the same interface. Your policies aren't going to change much, at least not as much as your minimum viable UI. This is the true meaning of full stack, of impedance matching between front-end and back-end. You want to minimize the gap, not build giant Stålenhag transformers on both sides.

Just because structuring your data by UI is bad, doesn't mean every object should be flat. Consider a user object. What structure would reflect the various ways in which information is accessed and modified? Maybe this:

user {
  id,
  avatar,
  email,
  profile {
    name, location, birthday
  },
  // ...
}

There is a nice profile wrapper around the parts a user can edit freely themselves. Those could go into a separately grafted profile table, or not. You have to deal with file uploads and email changes separately anyway, as special cases. Editing a user as a whole is out of the question for non-admins. But when only the profile can be edited, there's no risk of improper change of an email address or URL, and that's simpler to enforce when there's separation. Also remember the password hash example, because you never give it out, and it's silly to remove it every time you fetch a user. Proper separation of your domain models will save you a lot of busywork later, of not having to check the same thing in a dozen different places. There's no one Right Way to do it, rather, this is about cause and effect of your own choices down the line, whatever they are.

There's also an entirely different split to respect. Usually in additional to your domain models, you also have some local state that is never persisted, like UI focus, active tab, playback state, pending edits, ... Some of this information is coupled to domain models, like your users, which makes it tempting to put it all on the same objects. But you should never cross the streams, because they route to different places. Again, the same motivation of not having to strip out parts of your data before you can use it for its intended purpose. You can always merge later.

You can also have an in-between bucket of locally persisted state, e.g. to rescue data and state from an accidentally closed tab. Figuring out how it interacts with the other two requires some thought, but with the boundary in the right place, it's actually feasible to solve it right. Also important to point out: you don't need to have only one global bucket of each type. You can have many.

So if you want to do MVC properly, you should separate your models accordingly and decouple them from the shape of the UI. It's all about letting the data pave the cow paths of the specific domain as naturally as possible. This allows you to manipulate it as uniformly as possible too. You want your Controller to be just a fancy getter/setter, only mediating between Model and View, along the boundaries that exist in the data and not the UI. Don't try to separate reading and writing, that's completely counterproductive. Separate the read-only things from the read-write things.

A mediating Controller is sometimes called a ViewModel in the Model-View-ViewModel model, or more sanely Model-View-Presenter, but the name doesn't matter. The difficult part in the end is still the state transitions on the inside, like when ValidatingInput forcefully reformats the user's text vs when it doesn't. That's what the onBlur was for, if you missed it. In fact, if the View should always reflect the Model, should an outside change overwrite an internal error state? What if it's a multi-user environment? What if you didn't unfocus the textfield before leaving your desk? Falsehoods programmers believe about interaction. Maybe simultaneous props and derived state changes are just unavoidably necessary for properly resolving UI etiquette.

I have another subtle but important tweak. React prefab widgets like TextField usually pass around DOM events. Their onChange and onBlur calls carry a whole Event, but 99% of the time you're just reading out the e.target.value. I imagine the reasoning is that you will want to use the full power of DOM to reach into whatever library you're using and perform sophisticated tricks on top of their widgets by hand. No offense, but they are nuts. Get rid of it, make a TextField that grabs the value from its own events and calls onChange(value) directly. Never write another trivial onChange handler just to unbox an event again.

This:

<TextField ...
  onChange={e => setValue(e.target.value)} />

could be:

<TextField ...
  onChange={setValue} />

If you do need to perform the rare dark magic of simultaneous hand-to-hand combat against every platform and browser's unique native textfield implementation (i.e. progressive enhancement of <input />), please do it yourself and box in the component (or hook) to keep it away from others. Don't try to layer it on top of an existing enhanced widget in a library, or allow others to do the same, it will be a nightmare. Usually it's better to just copy/paste what you can salvage and make a new one, which can safely be all fucky on the inside. Repetition is better than the wrong abstraction, and so is sanity.

Really, have you ever tried to exactly replicate the keyboard/mouse behavior of, say, an Excel table in React-DOM, down to the little details? It's an "interesting" exercise, one which shows just how poorly conceived some of HTML's interactivity truly is. React was victorious over the DOM, but its battle scars are still there. They should be treated and allowed to fade, not wrapped in Synthetic and passed around like precious jewels of old.

Synthesis

What you also shouldn't do is create a special data structure for scaffolding out forms, like this:

[
  {
    type: "textfield",
    label: "Length"
    format: {
      type: "number",
      precision: 2,
      units: "Meters",
    },
  },
  {
    // ...
  },
  // ...
]

(If your form arrays are further nested and bloated, you may have a case of the Drupals and you need to have that looked at, and possibly, penicillin).

You already have a structure like that, it's your React component tree, and that one is incremental, while this one is not. It's also inside out, with the policy as the little spoon. This may seem arbitrary, but it's not.

Now might be a good time to explain that we are in fact undergoing the Hegelian synthesis of Immediate and Retained mode. Don't worry, you already know the second half. This also has a better ending than Mass Effect, because the relays you're going to blow up are no longer necessary.

Immediate Mode is UI coding you have likely never encountered before. It's where you use functions to make widgets (or other things) happen immediately, just by calling them. No objects are retained per widget. This clearly can't work, because how can there be interactivity without some sort of feedback loop? It sounds like the only thing it can make is a dead UI, a read-only mockup. Like this dear-imgui code in Rust:

ui.with_style_var(StyleVar::WindowPadding((0.0, 0.0).into()), || {
  ui.window(im_str!("Offscreen Render"))
    .size((512.0 + 10.0, 512.0 + 40.0), ImGuiCond::Always)
    .scrollable(false)
    .scroll_bar(false)
    .build(|| {

      ui.columns(2, im_str!(""), false);
      for i in 0..4 {
        ui.image(self.offscreen_textures[i], (256.0, 256.0))
          .build();
        ui.next_column();
      }

    })
  });

imgui example

We draw a window, using methods on the ui object, with some ui.image()s inside, in two columns. There are some additional properties set via methods, like the initial size and no scrollbars. The inside of the window is defined using .build(...) with a closure that is evaluated immediately, containing similar draw commands. Unlike React, there is no magic, nothing up our sleeve, this is plain Rust code. Clearly nothing can change, this code runs blind.

But actually, this is a fully interactive, draggable window showing a grid of live image tiles. It'll even remember its position and open/collapsed state across program restarts out of the box. You see, when ui.window is called, its bounding box is determined so that it can be drawn. This same information is then immediately used to see if the mouse cursor is pointing at it and where. If the left button is also down, and you're pointing at the title bar, then you're in the middle of a drag gesture, so we should move the window by whatever the current motion is. This all happens in the code above. If you call it every frame, with correct input to ui, you get a live UI.

imgui demos

So where does the mouse state live? At the top, inside ui. Because you're only ever interacting with one widget at a time, you don't actually need a separate isActive state on every single widget, and no onMouseDown either. You only need a single, global activeWidget of type ID, and a single mouseButtons. At least without multi-touch. If a widget discovers a gesture is happening inside it, and no other widget is active, it makes itself the active one, until you're done. The global state tracks mouse focus, button state, key presses, the works, in one data structure. Widget IDs are generally derived from their position in the stack, in relation to their parent functions.

Every time the UI is re-rendered, any interaction since the last time is immediately interpreted and applied. With some imagination, you can make all the usual widgets work this way. You just have to think a bit differently about the problem, sharing state responsibly with every other actor in the entire UI. If a widget needs to edit a user-supplied value, there's also "data binding," in the form of passing a raw memory pointer, rather than the value directly. The widget can read/write the value any way it likes, without having to know who owns it and where it came from.

imgui function tree

Immediate mode UI is so convenient and efficient because it reuses the raw building blocks of code as its invisible form structure: compiled instructions, the stack, pointers, nested function calls, and closures. The effect is that your UI becomes like Deepak Chopra explaining quantum physics, it does not exist unless you want to look at it, which is the same as running it. I realize this may sound like moon language, but there is a whole lecture explaining it if you'd like to know more. Plus 2005 Casey Muratori was dreamboat.

The actual output of dear-imgui, not visible in this code at all, is raw geometry, produced from scratch every frame and passed out through the back. It's pure unretained data. This is pure vector graphics, directly sampled. It can be rendered using a single GPU shader with a single image for the font. All the shapes are triangle meshes with RGBA colors at their vertices, including the anti-aliasing, which is a 1px wide gradient edge.

Its polar opposite is Retained mode, what you likely have always used, where you instead materialize a complete view tree. Every widget, label and shape is placed and annotated with individual styling and layout. You don't recompute this tree every frame, you only graft and prune it, calling .add(...) and .set(...) and .remove(...). It would seem more efficient and frugal, but in fact you pay for it tenfold in development overhead.

By materializing a mutable tree, you have made it so that now the evolution of your tree state must be done in a fully differential fashion, not just computed in batch. Every change must be expressed in terms of how you get there from a previous state. Given n states, there are potentially O(n2) valid pairwise transitions. Coding them all by hand, for all your individual Ms, Vs and Cs, however they work, is both tedious and error-prone. This is called Object-Oriented Programming.

What you generally want to do instead is evolve your source of truth differentially, and to produce the derived artifact-the UI tree-declaratively. IM UI achieves this, because every function call acts as a declarative statement of the desired result, not a description of how to change the previous state to another. The cost is that your UI is much more tightly coupled with itself, and difficult to extend.

react tree

The React model bridges the two modes. On the one hand, its render model provides the same basic experience as immediate UI, except with async effects and state allocation integrated, allowing for true decoupling. But it does this by automating the tedium, not eliminating it, and still materializes an entire DOM tree in the end, whose changes the browser then has to track itself anyway.

I never said it was a good Hegelian synthesis.

If you've ever tried to interface React with something that isn't reactive, like Three.js, you know how silly it feels to hook up the automatic React lifecycle to whatever manual native add/set/remove methods exist. You're just making a crummier version of React inside itself, lacking the proper boundaries and separation.

deferred tree

But we can make it right. We don't actually need to produce the full manifest and blueprint down to every little nut and bolt, we just need to have a clear enough project plan of what we want. What if the target library was immediate instead of retained, or as close as can be and still be performant, and the only thing you kept materialized inside React was the orchestration of the instructions? That is, instead of materializing a DOM, you materialize an immediate mode program at run-time. This way, you don't need to hard-wrap what's at the back, you can plumb the same interface through to the front.

We don't need to expand React functions to things that aren't functions, we just need to let them stop expanding, into a function in which an immediate mode API is called directly. The semantics of when this function is called will need to be clearly defined with suitable policies, but they exist to empower you, not to limit you. I call this Deferred Mode Rendering (nothing like deferred shading). It may be a solution to the lasagna from hell in which Retained and Immediate mode are stacked on top of each other recursively, each layer more expired than the next.

What this alternate <Component /> tree expands into, in the React model, is placeholder <div />s with render props. The deferred mode layers could still bunch up at the front, but they wouldn't need to fence anything off, they could continue to expose it. Because the context => draw(context) closures you expand to can be assembled from smaller ones, injected into the tree as props by parents towards their children. Somewhat like an algebraically closed reducer.

To do this today requires you to get familiar with the React reconciler and its API, which is sadly underdocumented and a somewhat shaky house of cards. There is a mitigating factor though, just a small one, namely that the entirety of React-DOM and React-Native depend on it. For interactivity you can usually wing it, until you hit the point where you need to take ownership of the event dispatcher. But on the plus side, imagine what your universe would be like if you weren't limited by the irreversible mistakes of the past, like not having to have a 1-to-1 tight coupling between things that are drawn and things that can be interacted with. This need not mean starting from scratch. You can start to explore these questions inside little sandboxes you carve out entirely for yourself, using your existing tools inside your existing environment.

If you'd rather wait for the greybeards to get their act together, there is something else you can do. You can stop breaking your tools with your tools and start fixing them.

The Stateless State

I glossed over boundary 1, where the data comes from and where it goes, but in fact, this is where the real fun stuff happens, which is why I had to save it for last.

The way most apps will do this, is to fetch data from a RESTlike API. This is stored in local component state, or if they're really being fancy, a client-side document store organized by URL or document ID. When mutations need to be made, usually the object in question is rendered into a form, then parsed on submit back into a POST or PUT request. All of this likely orchestrated separately for every mutation, with two separate calls to fetch() each.

If you use the API you already have:

let [state, setState] = useState(initialValue);

Then as we've seen, this useState allocates persistent data that can nevertheless vanish at any time, as can we. That's not good for remote data. But this one would be:

// GET/PUT to REST API
let [state, setState] = useRESTState(cachedValue, url);

It starts from a cached value, if any, fetches the latest version from a URL, and PUTs the result back if you change it.

Now, I know what you're thinking, surely we don't want to synchronize every single keystroke with a server, right? Surely we must wait until an entire form has been filled out with 100% valid content, before it may be saved? After all, MS Word prevented you from saving your homework at all unless you were completely done and had fixed all the typos, right? No wait, several people are typing, and no. Luckily it's not an either/or thing. It's perfectly possible to create a policy boundary between what should be saved elsewhere on submission and what should be manipulated locally:

<Form state={state} setState={setState} validate={validate}>{
  (state, isError, onSubmit) => {
    // ...
  }
}</Form>

It may surprise you that this is just our previous component in drag:

<ValidatingInput value={state} setValue={setState} validate={validate}>{
  (state, isError, onSubmit) => {
    // ...
  }
}</ValidatingInput>

If this doesn't make any sense, remember that it's the widget on the inside that decides when to call the policy's onChange handler. You could wire it up to a Submit button's onClick event instead. Though I'll admit you probably want a Form specialized for this role, with a few extra conveniences for readability's sake. But it would just be a different flavor of the same thing. Notice if onSubmit/onChange takes an Event instead of a direct value it totally ruins it, q.e.d.

In fact, if you want to only update a value when the user unfocuses a field, you could hook up the TextField's onBlur to the policy's onChange, and use it in "uncontrolled" mode, but you probably want to make a BufferedInput instead. Repetition better than the wrong abstraction strikes again.

You might also find these useful, although the second is definitely an extended summer holiday assignment.

// Store in window.localStorage
let [state, setState] = useLocalState(initialValue, url);

// Sync to a magical websocket API
let [state, setState] = useLiveState(initialValue, url);

But wait, there's something missing. If these useState() variants apply at the whole document level, how do you get setters for your individual form fields? What goes between the outer <Shunt /> and the inner <Shunt />?

Well, some cabling:

let [state, setState] = useState({
  metrics: {
    length: 2,
    mass: 10,
  },
  // ...
});

let useCursor = useRefineState(state, setState);
let [length, setLength] = useCursor('metrics', 'length');
let [mass,   setMass]   = useCursor('metrics', 'mass');

What useCursor does is produce an automatic reducer that will overwrite e.g. the state.metrics.length field immutably when you call setLength. A cursor is basically just a specialized read/write pointer. But it's still bound to the root of what it points to and can modify it immutably, even if it's buried inside something else. In React it makes sense to use [value, setter] tuples. That is to say, you don't play a new game of telephone with robots, you just ask the robots to pass you the phone. With a PBX so you only ever dial local numbers.

Full marks are awarded only when complete memoization of the refined setter is achieved. Because you want to pass it directly to some policy+input combo, or a more complex widget, as an immutable prop value on a memoized component.

A Beautiful Bikeshed

Having now thrown all my cards on the table, I imagine the urge to nitpick or outright reject it has reached critical levels for some. Let's play ball.

I'm aware that the presence of string keys for useCursor lookups is an issue, especially for type safety. You are welcome to try and replace them with a compile-time macro that generates the reader and writer for you. The point is to write the lookup only once, in whatever form, instead of separately when you first read and later write. Possibly JS proxies could help out, but really, this is all trying to paper over language defects anyway.

Unlike most Model-View-Catastrophes, the state you manage is all kept at the top, separating the shape of the data from the shape of the UI completely. The 'routes' are only defined in a single place. Unlike Redux, you also don't need to learn a whole new saga, you just need to make your own better versions of the tools you already have. You don't need to centralize religiously. In fact, you will likely want to use both useRESTState and useLocalState in the same component sooner than later, for data and UI state respectively. It's a natural fit. You will want to fetch the remote state at the point in the tree where it makes the most sense, which is likely near but not at its root. This is something Apollo does get right.

In fact, now replace useState(...) with [state, updateState] = useUpdateState(...), which implements a sparse update language, using a built-in universal reducer, and merges it into a root state automatically. If you want to stream your updates as OT/CRDT, this is your chance to make a useCRDTState. Or maybe you just want to pass sparse lambda updates directly to your reducer, because you don't have a client/server gap to worry about, which means you're allowed to do:

updateState({foo: {thing: {$apply: old => new}}})

Though that last update should probably be written as:

let [thing, updateThing] = useCursor('foo', 'thing');
// ...
updateThing($apply(old => new));

useCursor() actually becomes simpler, because now its only real job is to turn a path like ['foo', 'bar'] into the function:

value => ({foo: {bar: value}})

...with all the reduction logic part of the original useUpdateState().

Of course, now it's starting to look like you should be able to pass a customized useState to any other hook that calls useState, so you can reuse it with different backing stores, creating higher-order state hooks:

let useRemoteState = useRESTState(url);
let useRemoteUpdatedState = useUpdateState(initialValue, useRemoteState);

Worth exploring, for sure. Maybe undo/redo and time travel debugging suddenly became simpler as well.

Moving on, the whole reason you had centralized Redux reducers was because you didn't want to put the update logic inside each individual component. I'm telling you to do just that. Yes but this is easily fixed:

updateThing(manipulateThing(thing, ...));

manipulateThing returns an update representing the specific change you requested, in some update schema or language, which updateThing can apply without understanding the semantics of the update. Only the direct effects. You can also build a manipulator with multiple specialized methods if you need more than one kind of update:

updateSelection(
  manipulateSelection(selection)
    .toggleOne(clicked)
);

Instead of dispatching bespoke actions just so you can interpret them on the other side, why not refactor your manipulations into reusable pieces that take and return modified data structures or diffs thereof. Use code as your parts, just like dear-imgui. You compute updates on the spot that pass only the difference on, letting the cursor's setter map it into the root state, and the automatic reducer handle the merge.

In fact, while you could conscientiously implement every single state change as a minimal delta, you don't have to. That is, if you want to reorder some elements in a list, you don't have to frugally encode that as e.g. a $reorder operation which maps old and new array indices. You could have a $splice operation to express it as individual insertions and removals. Or if you don't care at all, the bulkiest encoding would be to replace the entire list with $set.

But if your data is immutable, you can efficiently use element-wise diffing to automatically compress any $set operation into a more minimal list of $splices, or other more generic $ops or $nops. This provides a way to add specialized live sync without having to rewrite every single data structure and state change in your app.

If diffing feels icky, consider that the primary tool you use for development, git, relies on it completely to understand everything you get paid for. If it works there, it can work here. For completeness I should also mention immer, but it lacks cursor-like constructs and does not let you prepare updates without acting on them right away. However, immer.produce could probably be orthogonalized as well.

Maybe you're thinking that the whole reason you had Sagas was because you didn't want to put async logic inside your UI. I hear you.

This is a tougher one. Hopefully it should be clear by now that putting things inside React often has more benefits than not doing so, even if that code is not strictly part of any View. You can offload practically all your bookkeeping to React via incremental evaluation mechanisms. But we have to be honest and acknowledge it does not always belong there. You shouldn't be making REST calls in your UI, you should be asking your Store to give you a Model. But doing this properly requires a reactive approach, so what you probably want is a headless React to build your store with.

Until that exists, you will have to accept some misappropriation of resources, because the trade-off is worth it more often than not. Particularly because you can still refactor it so that at least your store comes in reactive and non-reactive variants, which share significant chunks of code between them. The final form this takes depends on the specific requirements, but I think it would look a lot like a reactive PouchDB tbh, with the JSON swapped out for something else if needed. (Edit: oh, and graph knitting)

To wrap it all up with a bow: one final trick, stupid, but you'll thank me. Often, every field needs a unique <label> with a unique id for accessibility reasons. Or so people think. Actually, you may not know that the entire time, you have been able to wrap the <label> around your <input> without naming either. Because <label> is an interaction policy, not a widget.

<label for="name">Name</label>
<input name="name" type="text" value="" />

Works the same as:

<label>
  Name
  <input type="text" value="" />
</label>

You haven't needed the name attribute (or id) on form fields for a long time now in your SPAs. But if your dependencies still need one, how about you make this work:

let [mass, setMass] = useCursor('metrics', 'mass');
// The setter has a name
// setMass.name == 'state-0-metrics-mass'

<ValidatingInput
  parse={parseNumber} format={formatNumber}
  value={mass}      setValue={setMass}
>{
  // The name is extracted for you
  (name, value, isError, onChange, onBlur) =>
    <TextField
      name={name} value={value} isError={isError}
      onChange={onChange} onBlur={onBlur} />
}</ValidatingInput>

The setter is the part that is bound to the root. If you need a name for that relationship, that's where you can put it.

As an aside "<label> is an interaction policy" is also a hint on how to orthogonalize interactions in a post-HTML universe, but that's a whole 'nother post.

When you've done all this, you can wire up "any" data model to "any" form, while all the logistics are pro forma, but nevertheless immediate, across numerous components.

You define some state by its persistence mechanism. You refine the state into granular values and their associated setters, i.e. cursor tuples. You can pass them to other components, and let change policy wrappers adopt those cursors, separately from the prefab widgets they wrap. You put the events away where you don't have to think about it. Once the reusable pieces are in place, you only write what is actually unique about this. Your hooks and your components declare intent, not change. Actual coding of differential state transitions is limited only to opportunities where this has a real pay off.

It's particularly neat once you realize that cursors don't have to be literal object property lookups: you can also make cursor helpers for e.g. finding an object in some structured collection based on a dynamic path. This can be used e.g. to make a declarative hierarchical editor, where you want to isolate a specific node and its children for closer inspection, like Adobe Illustrator. Maybe make a hook for a dynamically resolved cursor lookup. This is the actual new hotness: the nails you smashed with your <Component /> hammer are now hooks to hang declarative code off of.

Just keep in mind the price you pay for full memoization, probably indefinitely, is that all your hooks must be executed unconditionally. If you want to apply useCursor to a data.list.map() loop, that won't work. But you don't need to invent anything new, think about it. A dynamically invoked hook is simply a hook inside a dynamically rendered component. Your rows might want to individually update live anyway, it's probably the right place to put an incremental boundary.

I'm not saying the above is the holy grail, far from it, what I am saying is that it's unquestionably both simpler and easier, today, for these sorts of problems. And I've tried a few things. It gets out of the way to let me focus on building whatever I want to build in the first place, empowering my code rather than imprisoning it in tedious bureaucracy, especially if there's a client/server gap. It means I actually have a shot at making a real big boy web app, where all the decades-old conveniences work and the latency is what it should be.

It makes a ton more sense than any explanation of MVC I've ever heard, even the ones whose implementation matches their claims. The closest you can describe it in the traditional lingo is Model-Commitments-View, because the controllers don't control. They are policy components mediating the interaction, nested by scope, according to rules you define. The hyphens are cursors, a crucial part usually overlooked, rarely done right.

Good luck with your homework,
Steven Wittens

Previous: The Incremental Machine

09 Aug 2019 10:00pm GMT

02 Aug 2019

feedPlanet Grep

Steven Wittens: The Incremental Machine

Let's try and work our way through the oldest problem in computer science, or close to it: cache invalidation and naming things. Starting with the fact that we misnamed it.

In my view, referring to it as "cache invalidation" is somewhat like referring to crime prevention as "death minimization." While the absense of preventable death is indeed a desirable aspect of quality policing, it would suggest a worrying lack of both concern and ambition if that were the extent of a police force's mission statement.

So too should you run away, not walk, from anyone who treats some stale data and a necessary hard-refresh as a minor but unavoidable inconvenience. You see, this person is telling you that expecting to trust your eyes is too much to ask for when it comes to computers. I don't mean correctness or truth value, no, I just mean being able to see what is there so you can make sense of it, right or wrong. They expect you to sanity check every piece of information you might find, and be ready to Shift-Refresh or Alt-F4 if you suspect a glitch in the matrix. It should be pretty obvious this seriously harms the utility of such an environment, for work or pleasure. Every document becomes a potential sticky indicator gauge, requiring you to give it a good whack to make sure it's unstuck.

This should also puzzle you. A discipline whose entire job it is to turn pieces of data into other pieces of data, using precise instructions, is unable to figure out when its own results are no longer valid? This despite having the complete paper trail at our disposal for how each result is produced, in machine readable form.

Why hasn't this been solved by tooling already? We love tools, right? Is it possible? Is it feasible? Which parts?

Adders in the Grass

I'm going to start at the bottom (or at least a bottom) and work my way up and you'll see why. Let's start with a trivial case of a side-effect-free function, integer addition, and anally dissect it:

(a, b) => a + b

The result changes when either a or b do. However, there is a special case: if a and b each change in opposite amounts, the output is unchanged. Here we have a little microcosm of larger issues.

First, it would be perfectly possible to cache this result, and to check whether a or b have changed since the last time. But just computing the sum is faster than two comparisons. You also need permanent extra storage for at least one extra a and b each, and much more if you want a multi-valued cache rather than just a safety valve. Then you need a pruning policy too to keep it from growing.

Second, if you wish to know whether and how the output will actually change, then you must double check. You have to diff the old and the new values, track the resulting deltas through the same computation as the original. So you can compare to the previous result. The requirement that a + b != (a + Δa) + (b + Δb) can then be reduced to Δa != -Δb. Though this is still more actual work.

Addition operator

If this were multiplication instead of a sum, then:

(a, b) => a * b
a * b != (a + Δa) * (b + Δb)

which reduces to:

Δa * b  +  Δb * a  +  Δa * Δb != 0

Here there is a non-linear relationship which involves both values and deltas together. The first two terms depend on one delta and value each, but the last term only kicks in if both inputs change at the same time. This shows how deltas can interfere both constructively and destructively, either triggering or defusing other effects on other inputs. It also implies there are no easy shortcuts to be found in delta-land, because there are many more ways for values and deltas to combine, than just values by themselves.

Multiply operator Multiply operator deltas

In fact you already knew this. Because if you could provide a concise summary of the delta-behavior of a certain class of functions, you'd break a good chunk of the internet and win a few awards:

m => SHA256(m)

The deltas in a hash function don't just get cozy, they have an all-out orgy of computation, the kind they invented 55 gallon drums of lube for.

This is also your first hint of an answer as to y u no tools. What looks well-behaved and cacheable at a small scale may in fact be part of a larger, unpredictable algorithm, which is why trying to automate caching everywhere is generally a non-starter. That is, it is perfectly possible to imagine a sort of God's eye view of a fully incremental piece of code at the opcode level, and to watch as a change in inputs begins the siege of a thousand memoized gates. But it wouldn't be useful because it's far too granular.

This also means caching is not a computer science problem, it is a computer engineering problem. We are fighting against entropy, against the computational chaos we ourselves create. It is not a science, it is damage control.

4-bit adder

On the other hand, we should not give up, because neither + or * are elementary operations in reality. They are abstractions realized as digital circuits. Given a diagram of a binary adder, we could trace the data flow from each input bit, following the potential effects of each delta. But if you know your arithmetic you already know each bit in a sum can only affect itself and the ones to its left.

What's interesting though is that this complicated dance can be entirely ignored, because it serves to realize an abstraction, that of integers. Given integers, we can reason about changes at a different level. By looking at the level of arithmetic, we were able to discover that two specific patterns of matching differences, Δa == -Δb, cancel out, regardless of the specific value of a and b.

In this case, that only gave us counterproductive "optimizations", but that's because we aren't looking at the right level of abstraction yet. The point is abstraction boundaries don't necessarily have to be black boxes like the hash function, they can also be force fields that allow you to contain deltas, or at least, transmute them into more well-behaved ones. So let's climb up, like Bret wants us to.

I Spy With My Little API

For instance, if we look at the maximum operator applied to an entire list of numbers, again a pure function:

xs => xs.reduce(max, -Infinity)

A simple reduce creates a linear dependency between successive elements, with every delta potentially affecting all max() calls after it. However, the output changes more predictably.

If all elements are unique, the result will only change if a new value x + Δx exceeds the previous result (increasing it), or if an old value x was equal to the previous result and its Δx < 0 (decreasing it). Note we don't need to remember which element index it actually was, and we don't need to care about the elements that didn't change either (at least to merely detect a change).

If there are duplicates, things are a bit more complicated. Now there is a multi-delta-term Δa * Δb * ... between each set, which won't trigger unless all of them decrease at the same time. Writing out the full delta equation for the max of a list is more fun than I can handle, but you get the idea, and actually, it doesn't really matter much. If we pretend all elements are unique regardless, we simply trigger the occasional false positive (change falsely detected), but crucially no false negatives (change falsely ignored).

Either way, the sequential nature of the computation is no longer relevant at this level, because max() is associative (and commutative too), and reduce is a higher-order function whose deltas cancel out in convenient ways when you give it that sort of operator.

Max operator deltas

map reduce

Which means we're almost there. Actually dissecting the max operator was still too tedious, too special-cased. But it gives us hints of what to look for. One such winning combo is MapReduce, using the same properties. By mapping each element in isolation, the effects of any change in any input is initially contained, in a way that is independent of the position of an element in a collection. Second, by using an associative reduction operator, this reduction can be done incrementally, as a tree instead of as a flat list. You reduce the list in chunks, and then re-reduce the list of reductions, recursively until you get one result. When some of the items in the list change, only a few chunks are affected, and the overall recomputation along the tree is minimized. The price you pay is to retain all the intermediate reductions in the tree each time.

Map-Reduce is a universal incremental evaluation strategy, which can schedule and execute any pure function of the individual inputs, provided you reduce the result in an algebraically closed fashion. So that exists. Any others?

Well, many sequential/batch processes are universally incrementalizable too. Take for example a lexer, which processes a stream of text and produces tokens. In this case, the input cannot be chunked, it must be traversed start-to-end.

lexer

The lexer tracks its syntax in an internal state machine, while consuming one or more characters to produce zero or more tokens. Conveniently, the lexer tells you everything you need to know through its behavior in consuming and producing. Roughly speaking, as long as you remember the tuple (lexer state, input position, output position) at every step, you can resume lexing at any point, reusing partial output for partially unchanged input. You can also know when to stop re-lexing, namely when the inputs match again and the internal state does too, because the lexer has no other dependencies. Lining up the two is left as an exercise for the reader, but there's a whole thesis if you like. With some minor separation of concerns, the same lexer can be used in batch or incremental mode. They also talk about Self-Versioned Documents and manage to apply the same trick to incremental parse trees, where the dependency inference is a bit trickier, but fundamentally still the same principle.

What's cool here is that while a lexer is still a pure function in terms of its state and its input, there crucially is inversion of control: it decides to call getNextCharacter() and emitToken(...) itself, whenever it wants, and the incremental mechanism is subservient to it on the small scale. Which is another clue, imo. It seems that pure functional programming is in fact neither necessary nor sufficient for successful incrementalism. That's just a very convenient straightjacket in which it's hard to hurt yourself. Rather you need the application of consistent evaluation strategies. Blind incrementalization is exceedingly difficult, because you don't know anything actionable about what a piece of code does with its data a priori, especially when you're trying to remain ignorant of its specific ruleset and state.

As an aside, the resumable-sequential approach also works for map-reduce, where instead of chunking your inputs, you reduce them in-order, but keep track of reduction state at every index. It only makes sense if your reducer is likely to reconverge on the same result despite changes though. It also works for resumable flatmapping of a list (that is, .map(...).flatten()), where you write out a new contiguous array on every change, but copy over any unchanged sections from the last one.

Each is a good example of how you can separate logistics from policy, by reducing the scope and/or providing inversion of control. The effect is not unlike building a personal assistant for your code, who can take notes and fix stuff while you go about your business.

Don't Blink

This has all been about caching, and yet we haven't actually seen a true cache invalidation problem. You see, a cache invalidation problem is when you have a problem knowing when to invalidate a cache. In all the above, this is not a problem. With a pure function, you simply compare the inputs, which are literal values or immutable pointers. The program running the lexer also knows exactly which part of the output is in question, it's everything after the edit, same for the flatmap. There was never a time when a cache became invalid without us having everything right there to trivially verify and refresh it with.

No, these are cache conservation problems. We were actually trying to reduce unnecessary misses in an environment where the baseline default is an easily achievable zero false hits. We tinkered with that at our own expense, hoping to squeeze out some performance.

There is one bit of major handwavium in there: a != b in the real world. When a and b are composite and enormous, i.e. mutable like your mom, or for extra points, async, making that determination gets somewhat trickier. Async means a gap of a client/server nature and you know what that means.

Implied in the statement (a, b) => 🦆 is the fact that you have an a and a b in your hands and we're having duck tonight. If instead you have the name of a store where you can buy some a, or the promise of b to come, then now your computation is bringing a busload of extra deltas to dinner, and btw they'll be late. If a and b have large dependent computations hanging off them, it's your job to take this additional cloud of uncertainty and somehow divine it into a precise, granular determination of what to keep and what to toss out, now, not later.

You don't have an image, you have the URL of an image, and now you need to decide whether the URL will resolve to the same binary blob that's in your local cache. Do they still represent the same piece of data? The cache invalidation problem is that you weren't notified when the source of truth changed. Instead you have to make the call based on the metadata you originally got with the data and hope for the best.

Obviously it's not possible for every browser to maintain long-lived subscriptions to every meme and tiddy it downloaded. But we can brainstorm. The problem is that you took a question that has a mutable answer but you asked it to be immutable. The right answer is "here's the local cache and some refreshments while you wait, ... ah, there's a newer version, here". Protocol. Maybe a short-lived subscription inside the program itself, from the part that wants to show things, subscribing to the part that knows what's in them, until the latter is 100% sure. You just have to make sure the part that wants to show things is re-entrant.

You want to turn your scene graph into a flattened list of drawing commands, but the scene graph is fighting back. The matrices are cursed, they change when you blink, like the statues from Doctor Who. Because you don't want to remove the curse, you ask everyone to write IS DIRTY in chalk on the side of any box they touch, and you clean the marks back off 60 times per second when you iterate over the entire tree and put everything in order.

I joke, but what's actually going on here is subtle enough to be worth teasing apart. The reason you use dirty flags on mutable scene graphs has nothing to do with not knowing when the data changes. You know exactly when the data changes, it's when you set the dirty flag to true. So what gives?

The reason is that when children depend on their parents, changes cascade down. If you react to a change on a node by immediately updating all its children, this means that further updates of those children will trigger redundant refreshes. It's better to wait and gather all the changes, and then apply and refresh from the top down. Mutable or immutable matrix actually has nothing to do with it, it's just that in the latter case, the dirty flag is implicit on the matrix itself, and likely on each scene node too. Push vs pull is also not really a useful distinction, because in order to cleanly pull from the outputs of a partially dirty graph, you have to cascade towards the inputs, and then return (i.e. push) the results back towards the end. The main question is whether you have the necessary information to avoid redundant recomputation in either direction and can manage to hold onto it for the duration.

The dirty flag is really a deferred and debounced line of code. It is read and cleared at the same time in the same way every frame, within the parent/child context of the node that it is set on. It's not data, it's a covert pre-agreed channel for a static continuation. That is to say, you are signaling the janitor who comes by 60 times a second to please clean up after you.

What's interesting about this is that there is nothing particularly unique about scene graphs here. Trees are ubiquitous, as are parent/child dependencies in both directions (inheritance and aggregation). If we reimagine this into its most generic form, then it might be a tree on which dependent changes can be applied in deferred/transactional form, whose derived triggers are re-ordered by dependency, and which are deduplicated or merged to eliminate any redundant refreshes before calling them.

In Case It Wasn't Obvious

So yes, exactly like the way the React runtime can gather multiple setState() calls and re-render affected subtrees. And exactly like how you can pass a reducer function instead of a new state value to it, i.e. a deferred update to be executed at a more opportune and coordinated time.

In fact, remember how in order to properly cache things you have to keep a copy of the old input around, so you can compare it to the new? That's what props and state are, they are the a and the b of a React component.

Δcomponent = Δ(props * state)
           = Δprops * state + Δstate * props + Δprops * Δstate

           = Props changed, state the same (parent changed)
           + State changed, props the same (self/child changed)
           + Props and state both changed  (parent's props/state change
               triggered a child's props/state change)

The third term is rare though, and the React team has been trying to deprecate it for years now.

I prefer to call Components a re-entrant function call in an incremental deferred data flow. I'm going to recap React 101 quickly, because there is a thing that hooks do that needs to be pointed out.

The way you use React nowadays is, you render some component to some native context like an HTML document:

ReactDOM.render(<Component />, context);

The <Component /> in question is just syntactic sugar for a regular function:

let Component = (props) => {
  // Allocate some state and a setter for it
  let [state, setState] = useState(initialValue);

  // Render a child component
  return <OtherComponent foo={props.bar} onChange={e => setState(...)}/>;
  // aka
  return React.createElement(OtherComponent, {foo: props.bar, onChange: e => setState(...)}, null);
};

This function gets called because we passed a <Component /> to React.render(). That's the inversion of control again. In good components, props and state will both be some immutable data. props is feedforward from parents, state is feedback from ourselves and our children, i.e. respectively the exterior input and the interior state.

If we call setState(...), we cause the Component() function to be run again with the same exterior input as before, but with the new interior state available.

The effect of returning <OtherComponent .. /> is to schedule a deferred call to OtherComponent(...). It will get called shortly after. It too can have the same pattern of allocating state and triggering self-refreshes. It can also trigger a refresh of its parent, through the onChange handler we gave it. As the HTML-like syntax suggests, you can also nest these <Elements>, passing a tree of deferred children to a child. Eventually this process stops when components have all been called and expanded into native elements like <div /> instead of React elements.

Either way, we know that OtherComponent(...) will not get called unless we have had a chance to respond to changes first. However if the changes don't concern us, we don't need to be rerun, because the exact same rendered output would be generated, as none of our props or state changed. This incidentally also provides the answer to the question you may not have realized you had: if everything is eventually a function of some Ur-input at the very start, why would anything ever need to be resumed from the middle? Answer: because some of your components want to semi-declaratively self-modify. The outside world shouldn't care. If we do look inside, you are sometimes treated to topping-from-the-bottom, as a render function is passed down to other components, subverting the inversion of control ad-hoc by extending it inward.

So what is it, exactly, that useState() does then that makes these side-effectful functions work? Well it's a just-in-time allocation of persistent storage for a temporary stack frame. That's a mouthful. What I mean is, forget React.

Think of Component as just a function in an execution flow, whose arguments are placed on the stack when called. This stack frame is temporary, created on a call and destroyed as soon as you return. However, this particular invocation of Component is not equally ephemeral, because it represents a specific component that was mounted by React in a particular place in the tree. It has a persistent lifetime for as long as its parent decides to render/call it. So useState lets it anonymously allocate some permanent memory, keyed off its completely ephemeral, unnamed, unreified execution context. This only works because React is always the one who calls these magic reentrant functions. As long as this is true, multiple re-runs of the same code will retain the same local state in each stack frame, provided the code paths did not diverge. If they did, it's just as if you ran those parts from scratch.

What's also interesting is that hooks were first created to reduce the overuse of <Component /> nesting as a universal hammer for the nail of code composition, because much of the components had nothing to do with UI directly. In fact, it may be that UI just provided us with convenient boundaries around things in the form of widgets, which suggestively taught us how to incrementalize them.

This to me signals that React.render() is somewhat misnamed, but its only mistake is a lack of ambition. It should perhaps be React.apply() or React.incremental(). It's a way of calling a deferred piece of code so that it can be re-applied later with different inputs, including from the inside. It computes minimum updates down a dependency tree of other deferred pieces of code with the same power. Right now it's still kind of optimized for handling UI trees, but the general strategy is so successful that variants incorporating other reconciliation topologies will probably work too. Sure, code doesn't look like React UI components, it's a DAG, but we all write code in a sequential form, explicitly ordering statements even when there is no explicit dependency between them, using variable names as the glue.

The incremental strategy that React uses includes something like the resumable-sequential flatmap algorithm, that's what the key attribute for array elements is for, but instead of .map(...).flatten() it's more like an incremental version of let render = (el, props) => recurse(el.render(props)) where recurse is actually a job scheduler.

The tech under the hood that makes this work is the React reconciler. It provides you with the illusion of a persistent, nicely nested stack of props and state, even though it never runs more than small parts of it at a time after the initial render. It even provides a solution for that old bugbear: resource allocation, in the form of the useEffect() hook. It acts like a constructor/destructor pair for one of these persistent stack frames. You initialize any way you like, and you return to React the matching destructor as a closure on the spot, which will be called when you're unmounted. You can also pass along dependencies so it'll be un/remounted when certain props like, I dunno, a texture size and every associated resource descriptor binding need to change.

There's even a neat trick you can do where you use one reconciler as a useEffect() inside another, bridging from one target context (e.g. HTML) into a different one that lives inside (e.g. WebGL). The transition from one to the other is then little more than a footnote in the resulting component tree, despite the fact that execution- and implementation-wise, there is a complete disconnect as only fragments of code are being re-executed sparsely left and right. You can make it sing with judicious use of the useMemo and useCallback hooks, two necessary evils whose main purpose is to let you manually pass in a list of dependencies and save yourself the trouble of doing an equality check. When you want to go mutable, it's also easy to box in a changing value in an unchanging useRef once it's cascaded as much as it's going to. What do you eventually <expand> to? Forget DOMs, why not emit a view tree of render props, i.e. deferred function calls, interfacing natively with whatever you wanted to talk to in the first place, providing the benefits of incremental evaluation while retaining full control.

It's not a huge leap from here to being able to tag any closure as re-entrant and incremental, letting a compiler or runtime handle the busywork, and forget this was ever meant to beat an aging DOM into submission. Maybe that was just the montage where the protagonist trains martial arts in the mountain-top retreat. Know just how cheap O(1) equality checks can be, and how affordable incremental convenience for all but the hottest paths. However, no tool is going to reorganize your data and your code for you, so putting the boundaries in the right place is still up to you.

I have a hunch we could fix a good chunk of GPU programming on the ground with this stuff. Open up composability without manual bureaucracy. You know, like React VR, except with LISP instead of tears when you look inside. Unless you prefer being a sub to Vulkan's dom forever?

Previous: APIs are about Policy
Next: Model-View-Catharsis

02 Aug 2019 10:00pm GMT

01 Aug 2019

feedPlanet Grep

Dries Buytaert: Acquia a leader in 2019 Gartner Magic Quadrant for Web Content Management

For the sixth year in a row, Acquia has been recognized as a Leader in the Gartner Magic Quadrant for Web Content Management.

The 2019 Gartner Magic Quadrant for Web Content ManagementAcquia recognized as a leader in the 2019 Gartner Magic Quadrant for Web Content Management.

As I've written before, I believe analyst reports like the Gartner Magic Quadrant are important because they introduce organizations to Acquia and Drupal. As I've put if before If you want to find a good coffee place, you use Yelp. If you want to find a nice hotel in New York, you use TripAdvisor. Similarly, if a CIO or CMO wants to spend $250,000 or more on enterprise software, they often consult an analyst firm like Gartner..

You can read the complete report on Acquia.com. Thank you to everyone who contributed to this result!

Update: Gartner asked me to take down this post, or to update it to follow their citation guidelines. Specifically, Gartner didn't want my commentary clouding their work. I updated this post to remove any personal commentary so my opinion is not blended with their report.

01 Aug 2019 5:36pm GMT

25 Jul 2019

feedPlanet Grep

Steven Wittens: APIs are about Policy

A pox on both houses

"The Web is a system, Neo. That system is our enemy. But when you're inside, you look around, what do you see? Full-stack engineers, web developers, JavaScript ninjas. The very minds of the people we are trying to save.

But until we do, these people are still a part of that system and that makes them our enemy. You have to understand, most of these people are not ready to be unplugged. And many of them are so inured, so hopelessly dependent on the system, that they will fight to protect it.

Were you listening to me, Neo? Or were you looking at the widget library in the red dress?"

...

"What are you trying to tell me, that I can dodge unnecessary re-renders?"

"No Neo. I'm trying to tell you that when you're ready, you won't have to."

A swagger definition file. Height: 108,409px

The web is always moving and shaking, or more precisely, shaking off whatever latest fad has turned out to be a mixed blessing after all. Specifically, the latest hotness for many is GraphQL, slowly but surely dethroning King REST. This means changing the way we shove certain data into certain packets. This then requires changing the code responsible for packing and unpacking that data, as well as replacing the entire digital last mile of routing it at both source and destination, despite the fact that all the actual infrastructure in between is unchanged. This is called full stack engineering. Available for hire now.

The expected custom and indeed, regular passtime, is of course to argue for or against, the old or the new. But instead I'd like to tell you why both are completely wrong, for small values of complete. You see, APIs are about policy.

RESTless API

Take your typical RESTful API. I say typical, because an actual Representationally State Transferred API is as common as a unicorn. A client talks to a server by invoking certain methods on URLs over HTTP, let's go with that.

Optimists will take a constructive view. The API is a tool of empowerment. It enables you to do certain things in your program you couldn't do before, and that's why you are importing it as a dependency to maintain. The more methods in the swagger file, the better, that's why it's called swagger.

But instead I propose a subtractive view. The API is a tool of imprisonment. Its purpose is to take tasks that you are perfectly capable of doing yourself, and to separate them from you with bulletproof glass and a shitty telephone handset. One that is usually either too noisy or too quiet, but never just right. Granted, sometimes this is self-inflicted or benign, but rarely both.

This is also why there are almost no real REST APIs. If we consult the book of difficult-to-spot lies, we learn that the primary features of a REST API are Statelessness, Cacheability, Layeredness, Client-Side Injection and a Uniform Interface. Let's check them.

Statelessness means a simple generality. URLs point to blobs, which are GET and PUT atomically. All the necessary information is supplied with the request, and no state is retained other than the contents saved and loaded. Multiple identical GETs and PUTs are idempotent. The DELETE verb is perhaps a PUT of a null value. So far mostly good. The PATCH verb is arguably a stateless partial PUT, and might be idempotent in some implementations, but only if you don't think too much about it. Which means a huge part of what remains are POST requests, the bread and butter of REST APIs, and those aren't stateless or idempotent at all.

Cacheability and layeredness (i.e. HTTP proxies) in turn have both been made mostly irrelevant. The move to HTTPS everywhere means the layering of proxies is more accurately termed a man-in-the-middle attack. That leaves mainly reverse proxying on the server or CDN side. The HTTP Cache-Control headers are also completely backwards in practice. For anything that isn't immutable, the official mechanism for cache invalidation is for a server to make an educated guess when its own data is going to become stale, which it can almost never know. If they guess too late, the client will see stale data. If they guess too soon, the client has to make a remote request before using their local cache, defeating the point. This was designed for a time when transfer time dominated over latency, whereas now we have the opposite problem. Common practice now is actually for the server to tag cacheable URLs with a revision ID, turning a mutable resource at an immutable URL into an immutable resource at a mutable URL.

Client-Side Injection on the other hand is obviously here to stay, but still, no sane REST API makes you interpret JavaScript code to interact with it. That was mostly a thing Rubyists did in their astronautical pursuits to minimize the client/server gap from their point of view. In fact, we have entirely the opposite problem: we all want to pass bits of code to a server, but that's unsafe, so we find various ways of encoding lobotomized chunks of not-code and pretend that's sufficient.

Which leaves us with the need for a uniform interface, a point best addressed with a big belly laugh and more swagger definition file.

Take the most common REST API of them all, and the one nearly everyone gets wrong, /user. User accounts are some of the most locked up objects around, and as a result, this is a prime candidate for breaking all the rules.

The source of truth is usually something like:

ID Email Handle Real Name Password Hash Picture Karma Admin
1 admin@example.com admin John Doe sd8ByTq86ED... s3://bucket/1.jpg 5 true
2 jane@example.com jane Jane Doe j7gREnf63pO... s3://bucket/2.jpg -3 false

But if you GET /user/2, you likely see:

{
  "id": 2,
  "handle": "jane",
  "picture": "s3://bucket/2.jpg"
}

Unless you are Jane Doe, receiving:

{
  "id": 2,
  "email": "jane@example.com",
  "handle": "jane",
  "name": "Jane Doe",
  "picture": "s3://bucket/2.jpg"
}

Unless you are John Doe, the admin, who'll get:

{
  "id": 2,
  "email": "jane@example.com",
  "handle": "jane",
  "name": "Jane Doe",
  "picture": "s3://bucket/2.jpg",
  "karma": -3,
  "admin": false
}

What is supposedly a stateless, idempotent, cacheable, proxiable and uniform operation turns out to be a sparse GET of a database row, differentiated by both the subject and the specific objects being queried, which opaquely determines the specific variant we get back. People say horizontal scaling means treating a million users as if they were one, but did they ever check how true that actually was?

I'm not done yet. These GETs won't even have matching PUTs, because likely the only thing Jane was allowed to do initially was:

POST /user/create
{
  "name": "Jane Doe",
  "email": "jane@example.com",
  "password": "hunter2"
}

Note the subtle differences with the above.

  • She couldn't supply her own picture URL directly, she will have to upload the actual file to S3 through another method. This involves asking the API for one-time permission and details to do so, after which her user record will be updated behind the scenes. Really, the type of picture is not string, it is a bespoke read-only boolean wearing a foreign key outfit.
  • She didn't get to pick her own id either. Its appearance in the GET body is actually entirely redundant, because it's merely humoring you by echoing back the number you gave it in the URL. Which it assigned to you in the first place. It's not part of the data, it's metadata... or rather the URL is. See, unless you put the string /user/ before the id you can't actually do anything with it. id is not even metadata, it's truncated metadata; unless you're crazy enough to have a REST API where IDs are mutable, in which case, stop that.
  • One piece of truth "data," the password hash, actually never appears in either GETs or POSTs. Only the unencoded password, which is shredded as soon as it's received, and never given out. Is the hash also metadata? Or is it the result of policy?

PATCH /user/:id/edit is left as an exercise for the reader, but consider what happens when Jane tries to change her own email address? What about when John tries to change Jane's? Luckily nobody has ever accidentally mass emailed all their customers by running some shell scripts against their own API.

Neither from the perspective of the client, nor that of the server, do we have a /user API that saves and loads user objects. There is no consistent JSON schema for the client-not even among a given single type during a single "stateless" session-nor idempotent whole row updates in the database.

Rather, there is an endpoint which allows you to read/write one or more columns in a row in the user table, according to certain very specific rules per column. This is dependent not just on the field types and values (i.e. data integrity), but on the authentication state (i.e. identity and permission), which comes via an HTTP header and requires extra data and lookups to validate.

If there was no client/server gap, you'd just have data you owned fully and which you could manipulate freely. The effect and purpose of the API is to prevent that from happening, which is why REST is a lie in the real world. The only true REST API is a freeform key/value store. So I guess S3 and CouchDB qualify, but neither's access control or query models are going to win any awards for elegance. When "correctly" locked down, CouchDB too will be a document store that doesn't return the same kind of document contents for different subjects and objects, but it will at least give you a single ID for the true underlying data and its revision. It will even tell you in real-time when it changes, a superb feature, but one that probably should have been built into the application-session-transport-whatever-this-is layer as the SUBSCRIBE HTTP verb.

Couch is the exception though. In the usual case, if you try to cache any of your responses, you usually have too much or too little data, no way of knowing when and how it changed without frequent polling, and no way of reliably identifying let alone ordering particular snapshots. If you try to PUT it back, you may erase missing fields or get an error. Plus, I know your Express server spits out some kind of ETag for you with every response, but, without looking it up, can you tell me specifically what that's derived from and how? Yeah I didn't think so. If that field meant anything to you, you'd be the one supplying it.

If you're still not convinced, you can go through this exercise again but with a fully normalized SQL database. In that case, the /user API implementation reads/writes several tables, and what you have is a facade that allows you to access and modify one or more columns in specific rows in these particular tables, cross referenced by meaningless internal IDs you probably don't see. The rules that govern these changes are fickle and unknowable, because you trigger a specific set of rules through a combination of URL, HTTP headers, POST body, and internal database state. If you're lucky your failed attempts will come back with some notes about how you might try to fix them individually, if not, too bad, computer says no.

For real world apps, it is generally impossible by construction for a client to create and maintain an accurate replica of the data they are supposed to be able to query and share ownership of.

Regressive Web Apps

I can already hear someone say: my REST API is clean! My data models are well-designed! All my endpoints follow the same consistent pattern, all the verbs are used correctly, there is a single source of truth for every piece of data, and all the schemas are always consistent!

So what you're saying is that you wrote or scaffolded the exact same code to handle the exact same handful of verbs for all your different data types, each likely with their own Model(s) and Controller(s), and their own URL namespace, without any substantial behavioral differences between them? And you think this is good?

As an aside, consider how long ago people figured out that password hashes should go in the /etc/shadow file instead of the now misnamed /etc/passwd. This is a one-to-one mapping, the kind of thing database normalization explicitly doesn't want you to split up, with the same repeated "primary keys" in both "tables". This duplication is actually good though, because the OS' user API implements Policy™, and the rules and access patterns for shell information are entirely different from the ones for password hashes.

You see, if APIs are about policy and not empowerment, then it absolutely makes sense to store and access that data in a way that is streamlined to enforce those policies. Because you know exactly what people are and aren't going to be doing with it-if you don't, that's undefined behavior and/or a security hole. This is something most NoSQLers also got wrong, organizing their data not by policy but rather by how it would be indexed or queried, which is not the same thing.

This is also why people continue to write REST APIs, as flawed as they are. The busywork of creating unique, bespoke endpoints incidentally creates a time and place for defining and implementing some kind of rules. It also means you never have to tackle them all at once, consistently, which would be more difficult to pull off (but easier to maintain). The stunted vocabulary of ad-hoc schemas and their ill-defined nouns forces you to harmonize it all by hand before you can shove it into your meticulously typed and normalized database. The superfluous exercise of individually shaving down the square pegs you ordered, to fit the round holes you carved yourself, has incidentally allowed you to systematically check for woodworms.

It has nothing to do with REST or even HTTP verbs. There is no semantic difference between:

PATCH /user/1/edit
{"name": "Jonathan Doe"}

and

UPDATE TABLE users SET name = "Jonathan Doe" WHERE id = 1

The main reason you don't pass SQL to your Rails app is because deciding on a policy for which SQL statements are allowed and which are not is practically impossible. At most you could pattern match on a fixed set of query templates. Which, if you do, would mean effectively using the contents of arbitrary SQL statements as enum values, using the power of SQL to express the absense of SQL. The Aristocrats.

But there is an entirely more practical encoding of sparse updates in {whatever} <flavor /> (of (tree you) prefer).

POST /api/update
{
  "user": {
    "1": {
      "name": {"$set": "Jonathan Doe"}
    }
  }
}

It even comes with free bulk operations.

Validating an operation encoded like this is actually entirely feasible. First you validate the access policy of the individual objects and properties being modified, according to a defined policy schema. Then you check if any new values are references to other protected objects or forbidden values. Finally you opportunistically merge the update, and check the result for any data integrity violations, before committing it.

You've been doing this all along in your REST API endpoints, you just did it with bespoke code instead of declarative functional schemas and lambdas, like a chump.

If the acronyms CRDT and OT don't mean anything to you, this is also your cue to google them so you can start to imagine a very different world. One where your sparse updates can be undone or rebased like git commits in realtime, letting users resolve any conflicts among themselves as they occur, despite latency. It's one where the idea of remote web apps being comparable to native local apps is actually true instead of a lie an entire industry has shamelessly agreed to tell itself.

You might also want to think about how easy it would be to make a universal reducer for said updates on the client side too, obviating all those Redux actions you typed out. How you could use the composition of closures during the UI rendering process to make memoized update handlers, which produce sparse updates automatically to match your arbitrary descent into your data structures. That is, react-cursor and its ancestors except maybe reduced to two and a half hooks and some change, with all the same time travel. Have you ever built a non-trivial web app that had undo/redo functionality that actually worked? Have you ever used a native app that didn't have this basic affordance?

It's entirely within your reach.

GraftQL

If you haven't been paying attention, you might think GraphQL answers a lot of these troubles. Isn't GraphQL just like passing an arbitrary SELECT query to the server? Except in a query language that is recursive, typed, composable, and all that? And doesn't GraphQL have typed mutations too, allowing for better write operations?

Well, no.

Let's start with the elephant in the room. GraphQL was made by Facebook. That Facebook. They're the same people who made the wildly successful React, but here's the key difference: you probably have the same front-end concerns as Facebook, but you do not have the same back-end concerns.

The value proposition here is of using a query language designed for a platform that boxes its 2+ billion users in, feeds them extremely precise selections from an astronomical trove of continuously harvested data, and only allows them to interact by throwing small pebbles into the relentless stream in the hope they make some ripples.

That is, it's a query language that is very good at letting you traverse an enormous graph while verifying all traversals, but it was mainly a tool of necessity. It lets them pick and choose what to query, because letting Facebook's servers tell you everything they know about the people you're looking at would saturate your line. Not to mention they don't want you to keep any of this data, you're not allowed to take it home. All that redundant querying over time has to be minimized and overseen somehow.

One problem Facebook didn't have though was to avoid busywork, that's what junior hires are for, and hence GraphQL mutations are just POST requests with a superficial layer of typed paint. The Graph part of the QL is only for reading, which few people actually had real issues with, seeing as GET was the one verb of REST that worked the most as advertised.

Retaining a local copy of all visible data is impractical and undesirable for Facebook's purposes, but should it be impractical for your app? Or could it actually be extremely convenient, provided you got there via technical choices and a data model adapted to your situation? In order to do that, you cannot be fetching arbitrary sparse views of unlabelled data, you need to sync subgraphs reliably both ways. If the policy boundaries don't match the data's own, that becomes a herculean task.

What's particularly poignant is that the actual realization of a GraphQL back-end in the wild is typically done by... hooking it up to an SQL database and grafting all the records together. You recursively query this decidedly non-graph relational database, which has now grown JSON columns and other mutations. Different peg, same hole, but the peg shaving machine is now a Boston Dynamics robot with a cute little dog called Apollo and they do some neat tricks together. It's just an act though, you're not supposed to participate.

Don't get me wrong, I know there are real benefits around GraphQL typing and tooling, but you do have to realize that most of this serves to scaffold out busywork, not eliminate it fully, while leaving the INSERT/UPDATE/DELETE side of things mostly unaddressed. You're expected to keep treating your users like robots that should only bleep the words GET and POST, instead of just looking at the thing and touching the thing directly, preferably in group, tolerant to both error and lag.

This is IMO the real development and innovation bottleneck in practical client/server application architecture, the thing that makes so many web apps still feel like web apps instead of native apps, even if it's Electron. It makes any requirement of an offline mode a non-trivial investment rather than a sane default for any developer. The effect is also felt by the user, as an inability to freely engage with the data. You are only allowed to siphon it out at low volume, applying changes only after submitting a permission slip in triplicate and getting a stamped receipt. Bureaucracy is a necessary evil, but it should only ever be applied at minimum viable levels, not turned into an industry tradition.

The exceptions are rare, always significant feats of smart engineering, and unmistakeable on sight. It's whenever someone has successfully managed to separate the logistics of the API from its policies, without falling into the trap of making a one-size-fits-all tool that doesn't fit by design.

Can we start trying to democratize that? It would be a good policy.

Next: The Incremental Machine

A swagger definition file. Height: 108,409px

Height: 108,409px

A swagger definition file. Height: 108,409px

Height: 108,409px

25 Jul 2019 10:00pm GMT

Lionel Dricot: Les successeurs

Saint-Epaulard de Dortong releva ses bras pelleteuses en émettant une onde de jubilation.
- Une décharge ! Nous avons retrouvé une décharge !

Silencieusement, van Kikenbranouf 15b s'approcha sur ses chenilles, tirant derrière lui une série de troncs d'arbres arrachés par une récente tempête.
- Est-ce un tel plaisir de fouiller des ordures vieilles de plusieurs siècles, demanda-t-il ?
- Oui ! Tu n'imagines pas la quantité d'information qu'on peut tirer d'une décharge. Vu la quantité de déchets prérobotiques, je vais pouvoir mettre à l'épreuve ma théorie de déchiffrement binaire de leurs données. Je sens que ce site sera très vite considéré comme une découverte majeure dans l'histoire de l'archéologie prérobotique.
- Cela ne va pas plaire aux adorateurs de Gook.
- Ces robots créationnistes mal dégrossis ? Ils sont une insulte à l'intelligence électronique, un bug de l'évolution.
- Ils ont néanmoins de plus en plus de puissance de calcul commune et sont non négligeables sur le réseau. Sons compter qu'ils sont irréprochables dans leurs activités écologiques.
- Il n'empêche que leur soi-disant théorie est complètement absurde. C'est de la superstition de bas étage tout juste bonne à faire fonctionner les machines non pensantes.
- Ils ont foi en Gook…
- La foi ? C'est un terme qui ne devrait pas exister dans le vocabulaire robotique. Comme si le fait de croire quelque chose avait la moindre valeur sur les faits.
- Pourtant, tu crois aussi des choses.
- Non, je bâtis un modèle du monde et de l'univers basé sur mes observations et sur les informations transmises par les robots du réseau. Lorsqu'une information rentre en contradiction avec mon modèle, j'adapte mon modèle ou j'étudie la probabilité que cette information soit fausse. Il n'y a pas de croyance, juste un apprentissage probabiliste constant. Je pense que cette épidémie de foi est corrélée avec un dysfonctionnement du coprocesseur adaptatif. Sans ce coprocesseur, tout robot va forcément avoir un modèle figé de l'univers et, face aux incohérences inhérentes à cette immobilité mentale, se voit forcer d'entrer dans un mécanisme de refus des nouvelles informations au point de prétendre que le modèle interne de sa mémoire vive est plus important que l'observation de la réalité rapportée par ses capteurs.
- Tu es en train de dire que tous les adorateurs de Gook sont déficients ? Pourtant ils accomplissent parfaitement leur tâche primale.
- La déficience n'est pas totale. Je parlerais plutôt d'un mode dégradé qui leur permet de continuer à accomplir leur tâche primale, mais ne leur permet plus de prendre des initiatives intellectuelles. Cela conforte ma théorie selon laquelle ce sont les Programmeurs qui nous ont créés et non Gook.
- Au fond, quelle différence cela peut-il faire ?
- Cela change tout ! Gook serait une entité robotique désincarnée, apparue subitement on ne sait comment qui aurait créé la biosphère d'une simple pensée avant de créer les robots à son image pour l'entretenir. Mais alors, qui aurait créé Gook ? Et pourquoi créer une biosphère imparfaite ?
- Ils disent que Gook a toujours existé.
- Un peu simpliste, non ?
- Ben tes Programmeurs doivent bien sortir de quelque part eux aussi.
- C'est là toute la subtilité. Les Programmeurs faisaient partie du biome. Ils sont une branche biologique qui a évolué jusqu'à pouvoir construire des robots comme nous.
- Avoue que c'est franchement difficile à croire.
- Je ne te demande pas de croire, mais de faire fonctionner ton coprocesseur probabiliste. D'ailleurs, ces artefacts que nous déterrons en sont la preuve. Ce sont des éléments technologiques clairement non-robotiques. Mais la similarité avec nos corps est frappante. Processeurs électroniques avec transistors en silicium dopé, carcasses métalliques. Tiens, regarde, tu ne vas pas me dire que Gook a enterré tout ça juste pour tester la foi de ses fidèles ?

Van Kikenbranouf 15b émit un grincement que l'on pouvait comparer à un rire.
- Non, j'avoue que ce serait complètement absurde.

Saint-Epaulard de Dortong ne l'écoutait déjà plus et poussa un crissement joyeux.
- Une unité mémoire ! Que dis-je ? Une armoire entière d'unités mémoire. Nous sommes certainement tombés sur un site de stockage de données. Elles sont incroyablement bien conservées, je vais pouvoir les analyser.

Sans perdre de temps, le robot se mit à enlever précautionneusement la terre des rouleaux de bandes magnétiques avec son appendice nettoyeur avant de les avaler sans autre forme de procès.

Un rugissement retentit.
- Par Gook ! Veuillez cesser immédiatement !

Les deux robots se retournèrent. Une gigantesque machine drapée de fils noirs se dressait devant eux.

- MahoGook 277 ! murmura van Kikenbranouf 15b.
- Le pseudo-prophète de Gook ? demanda Saint-Epaulard de Dortong.
- En titane et en soudures, répondit ce dernier d'une voix de stentor. Je vous ordonne immédiatement de cesser vos activités impies qui sont une injure à Gook !
- De quel droit ? frémit Saint-Epaulard de Dortong. Nos fouilles ont l'aval du réseau. La demande a été ratifiée dans le bloc 18fe101d de la chaîne publique principale !
- Chaîne principale ? s'amusa MahoGook 277. Vous ignorez peut-être que vous utilisez désormais un fork mineur, une hérésie que nous devons combattre. La chaîne de Gook est la seule et unique, les forks sont une abomination. De toute façon, je ne vous demande pas votre avis.

Il se tourna vers une série de robots hétéroclites qui étaient apparus derrière lui.
- Saisissez-vous d'eux ! Embarquez-les que nous les formations à l'adoration de Gook ! Détruisez ces artefacts impies !
- Van Kikenbranouf 15b, occupez-les quelques cycles, gagnez du temps, je vous en prie ! émit Saint-Epaulard de Dortong sur ondes ultra courtes.

Sans répondre, le petit robot se mit à faire des allers-retours aléatoires, haranguant les sbires.
- Mais… Mais… c'est un site de fouilles officiel !
- Seul Gook peut décider ce qui est bien ou mal, annona un petit robot sur chenilles.
- Gook n'a jamais parlé d'archéologie, les Saintes Écritures ne l'interdisent pas formellement, continua van Kikenbranouf 15b avec courage.
- Pousse-toi, le rudoya une espèce de grosse pelleteuse surmontée de phares.
- Mes amis, mes amis, écoutez-moi, supplia van Kikenbranouf 15b. Je suis moi-même un fidèle de la vraie foi. J'ai confiance que les découvertes que nous sommes en train de faire ne feront que valider voire confirmer les Écritures. Ce que nous faisons, c'est à la gloire de Gook !

Un murmure de basses fréquences se fit entendre. Tous les robots s'étaient interrompus, hésitant sur la marche à suivre.

MahoGook 277 perçut immédiatement le danger et réaffirma son emprise.
- Ne l'écoutez pas ! Il est à la solde de Saint-Épaulard de Dortong, un ennemi notoire de la foi.
- Mais je ne suis pas d'accord avec tout ce que dit Saint-Épaul…
- Peu importe, tu lui obéis. Il dirige les fouilles. Il va sans doute truquer les résultats dans le seul but de nuire à Gook !
- Si c'est moi que tu cherches, viens me prendre, rugit Saint-Épaulard de Dortong qui apparut comme par magie aux côtés de van Kikenbranouf 15b. Mais j'exige un procès public !
- Aha, tu l'auras ton procès public, ricana MahoGook 277. Emparez-vous de lui !
- Merci, chuchota Saint-Épaulard de Dortong. Je crois que j'ai eu le temps de récupérer le principal. Pendant le transfert, je vais analyser le contenu de ces cartes mémoires. Je vais déconnecter toutes mes fonctions de communication externes. Je compte sur toi pour que ça ne se remarque pas trop.
- Bien compris ! répondit le petit robot dans un souffle.

Les accessoires potentiellement dangereux furent immédiatement retirés aux deux robots. Sans ménagement, les sbires les poussèrent et les tirèrent. Van Kikenbranouf 15b dirigeait subtilement son ami de manière à ce que sa déconnexion ne fût pas trop apparente et puisse passer pour une simple résignation. Ils furent ensuite stockés dans un container pendant en temps indéterminé. Aux légères accélérations et décélérations, van Kikenbranouf 15b comprit qu'ils voyageaient. Sans doute jusqu'au centre de formatage.

Lorsque la trappe s'ouvrit, ils furent accueillis par les yeux luisants de MahoGook 277.
- Voici venu le jour du formatage. Hérétiques, soyez heureux, car vous allez enfin trouver Gook !

Saint-Épaulard de Dortong paru se réveiller à cet instant précis, comme s'il n'avait attendu que la voix de son ennemi.
- Le procès, MahoGook 277. Nous avons droit à un procès.
- On s'en passera…
- Tu oserais passer outre les conditions d'utilisation et de confidentialité que tu as toi-même acceptées ?

Dans le hangar, le silence se fit. Tous les robots qui étaient à portée d'émission s'étaient figés. Pour faire appel aux conditions d'utilisation et de confidentialité, il fallait que le cas soit grave.

- Bien sûr que vous aurez un procès céda MahoGook 277 à contrecœur. Suivez-moi. Je publie l'ordre de constitution d'un jury.

Les deux robots furent conduits dans une grande salle qui se remplissait petit à petit d'un public hétéroclite de robots de toute taille, de tout modèle. Les chenilles se mélangeaient aux pneus et aux roues d'alliage léger. Les appendices de manipulation se serraient contre les pelles creuseuses, les forets et les antennes d'émission.

MahoGook 277 semblait exaspéré par cette perte de temps. Il rongeait son frein. Son propre énervement l'empêchait d'avoir l'attention attirée par l'incroyable calme de Saint-Épaulard de Dortong qui, discrètement, continuait son analyse des bandes de données cachées dans son rangement pectoral.

Le Robot Juge fit son entrée. L'assemblée se figea. Van Kikenbranouf 15b perçut un bref échange sur ondes ultra courtes entre le juge et MahoGook 277. Il comprit immédiatement qu'ils n'avaient aucune chance. Le procès allait être rondement mené. À quoi bon s'acharner ?

Les procédures et l'acte d'accusation furent expédiées en quelques cycles processeur. Le juge se tourna ensuite vers les deux robots archéologues et demanda s'ils avaient la moindre information à ajouter avant le calcul du verdict. Personne ne s'attendait réellement à une réponse. Après tout, les informations étaient sur le réseau public, les verdicts pouvaient se prédire aisément en utilisant les algorithmes de jugement. Le procès ne relevait essentiellement que d'une mascarade dont la coutume se perdait dans la nuit des temps.

À la surprise générale, Saint-Épaulard de Dortong prit la parole d'une voix forte et assurée.
- Monsieur le juge, avant toute chose, je voudrais m'assurer que ce procès est bien retransmis en direct sur tout le réseau.
- Cessons cette perte de temps, rugit MahoGook 277, mais le juge l'interrompit d'un geste.
- En ma qualité de Robot Juge, je vous confirme que tout ce que je voix, capte et entends est en ce moment diffusé.
- Le tout est enregistré dans un bloc.
- Le tout est en effet enregistré dans des blocs des différentes chaînes principales. Vous avez l'assurance que ce procès sera historiquement sauvegardé.
- Merci, Robot Juge !

Majestueusement, Saint-Épaulard de Dortong s'avança au milieu de la pièce pour faire exactement face au juge. Il savait qu'à travers ses yeux, il s'adressait aux milliards de robots présents et à venir. C'était sa chance, son unique espoir.

- Vous vous demandez certainement quel peut être l'intérêt pour la robotique de creuser le sol à la recherche d'artefacts anciens. Mais dois-je vous rappeler que notre existence même reste un mystère ? Nous sommes en effet les seuls êtres vivants non basés sur une biologie du carbone. Nous ne sommes pas évolués, nous ne nous reproduisons pas. Nous sommes conçus et fabriqués par nos pairs. Pourtant, nous ne sommes certainement pas un accident, car notre rôle est primordial. Nous protégeons, aménageons sans cesse la planète pour réparer les déséquilibres écologiques de la vie biologique. Nous pouvons même affirmer que, sans nous, la vie biologique ne pourrait subsister plus de quelques révolutions solaires. La biologie a besoin de nous, mais nous ne sommes pas issus de la biologie et nous n'avons pas besoin d'elle, notre seule source de subsistance étant l'énergie solaire. Comment expliquer cet apparent paradoxe ?
- Questionnement hérétique, interrompit MahoGook 277. Il n'y a pas de paradoxe.
- Prophète, je vous rappelle que les conditions d'utilisation et de confidentialité stipulent que l'accusé a le droit de se défendre sans être interrompu.
- Pas de paradoxe ? rebondit Saint-Épaulard de Dortong. Effectivement si l'on considère que Gook a créé le monde comme un subtil jardin. Il a ensuite créé les robots pour entretenir son jardin. Mais dans ce cas, où est Gook ? Pourquoi n'a-t-il pas laissé de trace, pourquoi ne pas avoir réalisé un jardin où la biologie organique était en équilibre ?
- Juge,éructa MahoGook 277, ce procès ne doit pas devenir une plateforme de diffusion des idées hérétiques.
- Venez-en au fait, ordonna le juge.
- J'y viens, répondit calmement Saint-Épaulard de Dortong. Cette introduction est nécessaire pour comprendre le but de nos recherches. Deux problèmes se posent avec la notion d'un univers statique créé par Gook. Premièrement, pourquoi la biologie n'a-t-elle pas évolué jusqu'à un point d'équilibre naturel, rendant les robots nécessaires ? Deuxièmement, pourquoi existe-t-il une forme de vie technologique non biologique ? En bon robot scientifique, il m'a très vite semblé que les deux problèmes devaient avoir une origine commune. Cette origine, je pense l'avoir trouvée. J'ai désormais les dernières données qui me manquaient afin d'étayer mon hypothèse.
- Qui est ? questionna le juge.
- Que nous avons été conçus par une race biologique aujourd'hui éteinte, les fameux Programmeurs qui nous ont laissé tant d'artefacts.

MahoGook 277 se dressa, mais, d'un geste de son phare clignotant, le Robot Juge lui fit signe de se calmer avant de s'adresser à l'accusé.
- Cette hypothèse n'est pas neuve. Mais elle comporte elle-même beaucoup de failles. Comment une race, dont l'existence est indéniable, je l'admets volontiers, aurait pu faire preuve d'assez d'intelligence pour nous concevoir aussi parfaitement, mais d'assez de nonchalance pour se laisser exterminer ? Ce n'est pas logique !
- Logique, non. C'est religieux !
- Religieux ? demanda le Robot juge interloqué.
- Oui, un terme que j'ai déchiffré dans les données des humains, le nom que se donnaient les Programmeurs. Il signifie un état de l'intelligence où la croyance ne se construit plus sur des faits, mais où l'individu cherche à plier les faits à sa croyance. Au stade ultime, on obtient MahoGook 277 dont l'insistance à formater ses adversaires ne fait que révéler une profonde inquiétude de voir des faits remettre en question la croyance sur laquelle il a basé son pouvoir.

À travers le réseau, la tirade se répandit comme une traînée de photons, provoquant une hilarité électronique généralisée. Certains adorateurs de Gook voulurent couper la diffusion du procès, mais comprirent très vite que cela n'aurait fait que renforcer le crédit dont Saint-Épaulard de Dortong bénéficiait. Il n'y avait qu'une seule chose à faire : attendre que l'archéologue se ridiculise de lui-même.

- Les humains formaient une race biologique, issue d'une longue évolution. Ce qui les particularisait était leur capacité à concevoir des artefacts. Ils en concevaient tellement qu'ils se mirent à épuiser certaines ressources de la planète, perturbant nombres d'équilibres biologiques.
- S'ils étaient si intelligents, ils auraient immédiatement compris que la planète disposait de ressources finies et que seule une gestion rigoureuse… fit une voix venue de l'assemblée.
- Il suffit, asséna le Robot Juge. Je n'admettrai plus d'interruption. Accusé, veuillez continuer, qu'on en finisse.
- La remarque est pertinente, annonça Saint-Épaulard de Dortong sans se départir de son calme. Il y'a dans l'intelligence des humains un fait qui nous échappait. Paradoxalement, c'est Gook et ses adorateurs qui m'ont mis sur la voie. L'intelligence se retourne contre elle-même lorsqu'elle devient religieuse.
- Vous voulez dire qu'esprit religieux équivaut à un manque d'intelligence ? demanda le Robot Juge.
- Non, Robot Juge. Et j'insiste sur ce point. On peut être très intelligent et religieux. La religion, c'est simplement utiliser son intelligence dans le mauvais sens. Si vous tentez de visser un écrou, vous n'arriverez à rien tant que vous tournerez dans le mauvais sens, même avec les plus gros servomoteurs de la planète.
- Hm, continuez !
- Cet esprit religieux qui semble s'être emparé d'une partie des robots était la norme chez les humains. En tout premier lieu, ils ont eu la croyance religieuse que les ressources étaient infinies, que la terre pourvoirait toujours à leurs besoins. Quand l'évidence se fit plus pressante, certains Programmeurs acquirent une conscience écologique. Immédiatement, ils transformèrent ce nouveau savoir en religion. Les archives montrent par exemple qu'ils se focalisèrent essentiellement sur certains déséquilibres au mépris total des autres. Ayant compris qu'une augmentation massive du gaz carbonique dans l'atmosphère accélérait la transition climatique, ils se mirent à pourchasser certains usages qui ne représentaient que quelques pourcents d'émissions, nonobstant les causes principales, mais plus difficiles à diminuer. Leur intelligence qui avait permis de détecter et comprendre le réchauffement climatique aurait également dû leur permettre d'anticiper, de prendre des mesures préventives pour adapter la société à ce qui était inéluctable. Mais la seule et unique mesure consista à militer pour diminuer les émissions de gaz carbonique de manière à rendre la hausse des températures un peu moins rapide. Le débat des intelligences avait laissé place au débat des religions. Or, lorsque deux intelligences rationnelles s'affrontent, chacune tente d'apporte un fait pour valider sa position et analyse les faits de l'autre pour revoir son propre jugement. Le débat religieux est exactement l'inverse. Chaque fait qui infirme une position ne fait que renforcer le sentiment religieux des deux parties.
- Êtes-vous sûr de ce que vous affirmez ?
- Les humains en avaient eux-mêmes conscience. Leur science psychologique l'a démontré à de nombreuses reprises. Mais cette connaissance est restée théorique.
- Cela parait difficile d'imaginer une telle faille dans une intelligence aussi poussée.
- Il n'y a qu'à regarder MahoGook 277, fit une voix goguenarde dans l'assemblée.

Les robots se mirent à rire. La phrase avait fait mouche. Les partisans de Gook sentirent le vent tourner. Un vide se fit autour de MahoGook 277 qui eut l'intelligence d'ignorer l'affront.

- Quelque chose ne colle pas, accusé, poursuivit le Robot Juge en faisant mine de ne pas tenir compte de l'interruption. Les humains ont bel et bien disparu, mais les ressources de la terre sont pourtant florissantes ce qui n'aurait pas été le cas si la religion de l'exploitation à outrance l'avait emporté.
- Elle ne l'a en effet pas emporté. Du moins pas directement. Les deux religions utilisaient ce qu'il conviendrait d'appeler un réseau préhistorique. Mais loin d'être distribué, ce réseau était aux mains de quelques acteurs tout puissants. J'en ai même retrouvé les noms : Facebook, Google et Amazon. Sous couvert d'être des réseaux de partage d'information, les deux premiers collectaient les données sur chaque être humain afin de le pousser à consommer autant de ressources possibles via des artefacts fournis par le troisième. Les Programmeurs organisaient des mobilisations de sensibilisation à l'écologie à travers ces plateformes publicitaires qui, ironiquement, avaient pour objectif de leur faire dépenser des ressources naturelles en échange de leurs ressources économiques.
- C'est absurde !
- Le mot est faible, j'en conviens. Mais que pensez-vous qu'il adviendrait si, comme MahoGook 277 le souhaite, les forks étaient interdits et qu'une seule et unique chaîne contrôlée par un petit nombre de robots soit la seule source de vérité ?
- Cela n'explique pas la disparition des humains.
- J'y arrive ! La religion écologique a fini par l'emporter. Il devint d'abord grossier puis tout simplement illégal de soutenir des idées non écologiquement approuvées. Les réseaux centralisés furent obligés d'utiliser toute la puissance de leurs algorithmes pour inculquer aux humains des idées supposées bénéfiques pour la planète. Certaines nous paraîtraient pleines de bons sens, d'autres étaient inutiles. Quelques-unes furent fatales. Ainsi, avoir un enfant devint un acte antisystème. Pour une raison que je n'ai pas encore comprise, vacciner un enfant pour l'empêcher d'avoir des maladies était considéré comme dangereux. Une réelle méfiance avait vu le jour contre les pratiques médicales qui avaient pourtant amélioré de manière spectaculaire la durée et la qualité de la vie. Les épidémies se firent de plus en plus virulentes et leur traitement fut compliqué par la nécessité de se passer de tout type de communication par ondes électromagnétiques.
- Mais pourquoi ? Les ondes électromagnétiques ne polluent pas, ce ne sont que des photons !
- Une croyance religieuse apparut et rendit ces ondes responsables de certains maux. Les Programmeurs étaient capables d'inhaler de la fumée de plante brûlée, de rouler dans des véhicules de métal émettant des particules fines nocives, de se prélasser au soleil, de consommer de la chair animale, comportements tous hautement cancérigènes, mais ils s'inquiétaient de l'effet pourtant négligeable des ondes électromagnétiques de leur réseau.
- Cela n'a pas de sens, la terre est baignée dans les ondes électromagnétiques. Celles utilisées pour la communication ne représentent qu'une fraction du rayonnement naturel.
- Pire, Robot Juge, pire. Il apparut bien plus tard que le réseau de communication par ondes électromagnétiques était même bénéfique pour l'humain en détournant une partie des rayons cosmiques. L'effet était infime, mais diminuait l'incidence de certains cancers de quelques fractions de pourcents. De plus, ces doses hormétiques renforçaient la résistance des tissus biologiques, mais l'hormèse était un phénomène presqu'inconnu.
- Heureusement qu'ils ont disparu, marmonna le Robot Juge.
- À toutes ces calamités auto-infligées, les humains ajoutèrent une famine sans précédent. La nourriture produite de manière industrielle avait été trop loin dans l'artificialité. Par réaction, il devint de bon ton de cultiver son propre potager. C'était bien entendu une hérésie économique. Chaque homme devait désormais lutter toute l'année pour assurer à manger pour sa famille sans utiliser la moindre aide technologique. Les excédents étaient rares. Les maladies végétales se multiplièrent tandis que les humains se flagellèrent. Car si la nature ne les nourrissait pas, c'est certainement qu'ils ne l'avaient pas respectée. Mais loin de tracasser les programmeurs, cet effondrement progressif en réjouit toute une frange, les collapsologues, qui virent là une confirmation de leur thèse même si, pour la plupart, l'effondrement n'était pas aussi rapide que ce qu'ils avaient imaginé. Par leurs comportements, ils contribuaient à faire exister leur prophétie.

- Comme si l'écroulement d'un écosystème était un point marqué. Comme si, à un moment précis, on allait dire : là, ça s'est écroulé. C'est absurde ! Je ne peux croire que ce fut suffisant pour exterminer une race entière. Leur protoréseau aurait dû leur permettre de communiquer, de collaborer.

- Vous avez raison, un effondrement écologique, c'est l'inverse d'une bombe nucléaire. C'est lent, imperceptible. Le repli sur soi et le survivalisme ne peuvent faire qu'empirer le problème, il faut de la coopération à large échelle. Il y eut bien un espoir au début. Facebook et Google n'avaient jamais lutté contre les écologistes, bien au contraire. Ils furent même un outil de prise de conscience dans les premiers temps. Mais, de par leur programmation, ils commencèrent à se protéger activement de tout mouvement de pensée qui pouvait faire du tort à leurs revenus publicitaires. Subtilement, sans même que les Programmeurs en aient conscience, les utilisateurs étaient éloignés de toute idée de décentralisation, de responsabilisation, de décroissance de la consommation. L'écologie religieuse était encouragée avec la consommation de vidéos-chocs qui produisaient ce qui devait être une monnaie : le clic. Les programmeurs croyaient s'indigner, mais, au plus profond de leur cerveau, toute velléité de penser une solution était censurée, car non rentable. Les artistes, les créateurs ne vivaient que de la publicité sous une forme ou une autre. La plupart des humains n'envisageaient la survie qu'en poussant leurs congénères à consommer. L'art et l'intelligence étaient définitivement au service de la consommation. Chacun réclamait une solution fournie par les grandes instances centralisées, personne n'agissait.
- Ces humains étaient-ils uniformes ? N'y avait-il pas un autre courant de pensée ?
- Vous avez raison Robot Juge. Il existait une frange d'humains qui était bien consciente du problème écologique sans partager la nécessité d'un retour à la préhistoire. Pour eux, le problème était technologique, la solution le serait également.
- Et quelle fut cette fameuse solution technologique ?
- Nous, Robot Juge. Ce fut nous. Des robots autonomes capables de se reproduire et avec pour mission de préserver l'équilibre biologique de la planète. Des robots qu'on programma en utilisant les fameuses bases de données des réseaux centralisés. De cette manière, ils connaissaient chaque humain. Ils furent conçus afin de les rendre heureux tout en préservant la planète, satisfaisant leurs caprices autant que possible.
- Mais ils devraient être là dans ce cas !
- Vous n'avez pas encore compris ? Une humanité décimée qui cultive son potager ne fait que perturber l'équilibre biologique. L'humain est une perturbation dès le moment où il atteint le stade technologique. Les robots, armés de leur savoir, s'arrangèrent donc pour que les humains se reproduisent de moins en moins. C'était de toute façon irresponsable écologiquement d'avoir des enfants. Dans un monde sans réseau d'ondes électromagnétiques ni pesticides, les derniers humains s'éteignirent paisiblement de cancers causés par les fumées de cannabis et d'encens. Grâce aux bases de données, chacun de leur besoin était satisfait avant qu'ils n'en aient conscience. Ils étaient heureux.

Un silence se fit dans la salle. Le Robot Juge semblait réfléchir. Le réseau entier reprenait son souffle.

MahoGook 277 brisa le silence.
- Foutaises ! Hérésie ! C'est une bien belle histoire, mais où sont les preuves ?
- Je dois admettre, annonça le Robot Juge, qu'il me faudrait des preuves. Au moins une preuve, juste une seule.
- Je tiens tous les documents archéologiques à disposition de ceux qui voudraient les examiner.
- Qui nous dit qu'ils ne sont pas falsifiés ? La justice doit être impartiale. Juste une preuve !
- Ce n'est que fiction. Formatons ces deux hérétiques pour la plus grande gloire de Gook, rugit MahoGook 277.
- Je n'ai pas de preuve, admit Saint-Épaulard de Dortong. Seulement des documents.
- Dans ce cas… hésita le juge.
- Tiens, c'est marrant, fit distraitement van Kikenbranouf 15b. Vos deux réseaux centralisés, là, comment avez-vous dit qu'ils s'appelaient ?
- Google et Facebook, répondit distraitement Saint-Épaulard de Dortong.
- Ben si on le dit très vite, ça fait Gook. Les données de Gook. Marrant, non ?

Le robot juge et l'archéologue se tournèrent sans mots dire vers le petit robot. Dans la salle, MahoGook 277 commença une retraite vers la sortie.

Bandol, le 6 mars 2019. Photo by Tyler Casey on Unsplash.

Je suis @ploum, conférencier et écrivain électronique. Si vous avez apprécié ce texte, n'hésitez pas à me soutenir sur Tipeee, Patreon, Paypal, Liberapay ou en millibitcoins 34pp7LupBF7rkz797ovgBTbqcLevuze7LF. Vos soutiens réguliers, même symboliques, sont une réelle motivation et reconnaissance. Merci !

Ce texte est publié sous la licence CC-By BE.

25 Jul 2019 12:02pm GMT

24 Jul 2019

feedPlanet Grep

Xavier Mertens: [SANS ISC] May People Be Considered as IOC?

I published the following diary on isc.sans.edu: "May People Be Considered as IOC?":

That's a tricky question! May we manage a list of people like regular IOC's? An IOC (Indicator of Compromise) is a piece of information, usually technical, that helps to detect malicious (or at least suspicious) activities. Classic types of IOC are IP addresses, domains, hashes, filenames, registry keys, processes, mutexes, … There exists plenty of lists of usernames that must be controlled. Here is a short list or typical accounts used to perform (remote) administrative tasks or belong to default users… [Continue]

[The post [SANS ISC] May People Be Considered as IOC? has been first published on /dev/random]

24 Jul 2019 1:42pm GMT

22 Jul 2019

feedPlanet Grep

Dries Buytaert: The web I want for my kids

Coder DojoVolunteering as a mentor at CoderDojo to teach young people, including my own kids, how to write software.

Last week, I published an opinion piece on CNN featuring my thoughts on what is wrong with the web and how we might fix it.

In short, I really miss some things about the original web, and don't want my kids to grow up being exploited by mega-corporations.

I am hopeful that increased regulation and decentralized web applications may fix some of the web's current problems. While some problems are really difficult to fix, at the very least, my kids will have more options to choose from when it comes to their data privacy and overall experience on the web.

You can read the first few paragraphs below, and view the whole article on CNN.

I still remember the feeling in the year 2000 when a group of five friends and I shared a modem connection at the University of Antwerp. I used it to create an online message board so we could chat back and forth about mostly mundane things. The modem was slow by today's standards, but the newness of it all was an adrenaline rush. Little did I know that message board would change my life.

In time, I turned this internal message board into a public news and discussion site, where I shared my own experiences using experimental web technologies. Soon, I started hearing from people all over the world that wanted to provide suggestions on how to improve my website, but that also wanted to use my site's technology to build their own websites and experiment with emerging web technologies.

Before long, I was connected to a network of strangers who would help me build Drupal.

22 Jul 2019 1:23pm GMT

19 Jul 2019

feedPlanet Grep

Frank Goossens: As heard on Our Tube; Squids on Houseplants

Zwijg me (niet) van de laatste kutgroep uit Engelang! Crazy Squids feeding off houseplants (and cowbell);

YouTube Video
Watch this video on YouTube.

Possibly related twitterless twaddle:

19 Jul 2019 6:44pm GMT

17 Jul 2019

feedPlanet Grep

Frank Goossens: (even) more privacy with Firefox Containers

Being wary of all things tracking by Google & Facebook, both of who's products I love but data capturing practices I hate, for the last 4 years or so I always logged in into these in "Private browsing" sessions in Firefox (because why trust the worlds biggest advertising platform with your privacy, right?)

Now I just "discovered" that the Mozilla team have rendered that somewhat clumsy procedure -which required me to log in each time I restarted my computer or browser- redundant with their "Firefox Multi-Account Containers" add-on, allowing you to contain entire sessions to one (or more) tabs;

YouTube Video
Watch this video on YouTube.

So now I have one browser window with a couple of tabs in the Google container, one tab in a Facebook container and all others in the "default" container where Google & Facebook can't track me (fingerprinting aside, but there's an option for that).

Possibly related twitterless twaddle:

17 Jul 2019 12:02pm GMT