I built the same concurrency library in Go and Python, two languages, totally different ergonomics

Posted by kwargs_@reddit | programming | View on Reddit | 27 comments

I’ve been obsessed with making concurrency ergonomic for a few years now.

I wrote the same fan-out/fan-in pipeline library twice:

gliter (Go) - goroutines, channels, work pools, and simple composition
pipevine (Python) - async + multiprocessing with operator overloading for more fluent chaining

Both solve the same problems (retries, backpressure, parallel enrichment, fan-in merges) but the experience of writing and reading them couldn’t be more different.

Go feels explicit, stable, and correct by design.
Python feels fluid, expressive, but harder to make bulletproof.

Curious what people think: do we actually want concurrency to be ergonomic, or is some friction a necessary guardrail?

(I’ll drop links to both repos and examples in the first comment.)

[-]

somebodddy@reddit

Go feels explicit, stable, and correct by design. Python feels fluid, expressive, but harder to make bulletproof.

Can you elaborate on what in the Go library makes it "correct by design"? If it's the static typing, then:

You kind of messed that up in Go, because I've looked at your code and all your combinators are taking *Pipeline[T] and returning *Pipeline[T]. The exact same T. By enforcing a single type for the entire pipleine you prevent much of the advantage static typing brings because users will be forced to use a universal type instead of "evolving" a type through the pipeline.
Python also has generics which can be used to "enforce" static typing on the functions that participate in the pipeline. It'd be enforced by third party type checkers, not by Python itself, but it's still much more than you currently do, and using them (properly!) will make your library more bullet...resistant.

[-]

I don't think it's possible to build the API you've described in Go with different generics per stage. From what I remember, the problem is when you try to add a new generic variable on a struct method that is not also defined on the struct. I tried and wasn't able to find a solution so I settled for the current design. If you can make it work, please share.

[-]

somebodddy@reddit

You can have your internal lists of functions use func(any) (any, error) and do the conversion inside your library code, while the Pipeline type itself has two generic parameters - S for the input of the first stage and T for the output of the last stage. It'd make your library code more complex, but have the benefit of your users' code being simpler.

I recognize that Go has made a decision to push complexity outside (language -> libraries -> user code), but given that it’s a stupid-ass decision - I’ve elected to ignore it.

[-]

Nekuromento@reddit

Go correct by design

I'm not sure how this is possible, especially in concurrency related parts of go.

Are you impressed by channel type safety?

Just curious, have you ran into issues with closed/null channels yet? Do you feel like error propagation and especially panic propagation is 'correct by design' or 'ergonomic'?

[-]

kwargs_@reddit (OP)

"correct by design" may have been a bit hyperbolic but the pieces did snap together much easier in Go. Closed/nil channels haven't really burned me yet but deadlocks and memory leaks have though thats more of a composition/logic problem.

error propagation in go ergonomic? no. but extremely stable. By contrast, I hate that I can't know a python/javascript function throws by just inspecting the function signature.

[-]

kwargs_@reddit (OP)

- Go version → github.com/arrno/gliter
- Python version → github.com/arrno/pipevine


```Go
gliter.NewPipeline(exampleGen()).
    WorkPool(
        func(item int) (int, error) { return 1 + item, nil },
        3, //
 numWorkers
        WithBuffer(6),
        WithRetry(2),
    ).
    WorkPool(
        func(item int) (int, error) { return 2 + item, nil },
        6, //
 numWorkers
        WithBuffer(12),
        WithRetry(2),
    ).
    Run()
```

VS

```Python
(buffer=10, retries=3, num_workers=4)
async def process_data(item, state):
    #
 Your processing logic here
    return item * 2


u/work_pool(retries=2, num_workers=3, multi_proc=True)
async def validate_data(item, state):
    if item < 0:
        raise ValueError("Negative values not allowed")
    return item


#
 Create and run pipeline
result = await (
    Pipeline(range(100)) >> 
    process_data >> 
    validate_data
).run()
```

[-]

somebodddy@reddit

Trying to modify your Python version to look more like your Go version:

result = await Pipeline(range(100))
    .work_pool(
        process_data,
        buffer=10,
        retries=3,
        num_workers=4,
    ).work_pool(
        validate_data,
        retries=2,
        num_workers=3,
        multi_proc=True,
    ).run()

I don't think it makes it less fluid?

[-]

kwargs_@reddit (OP)

yea the python API also has:

pipe = (Pipeline(data_source)
.stage(preprocessing_stage)
.stage(analysis_stage)
.stage(output_stage))
result = await pipe.run()

but I find the syntactic sugary versions hard to resist. No sugar in Go though..

[-]

mr_birkenblatt@reddit

The syntactic sugar is great and all but to a person that isn't familiar with your library it's extremely hard to figure out what's going on. And the operator overloading doesn't make it easy to go to definition

[-]

guepier@reddit

Having an API that’s so intuitive that you can read it without knowing it is definitely nice, but it’s not essential.

Some domains are complex, and cannot fully hide this complexity. Having an API that reflects this can be fine. In the same vein, having domain-specific syntax can be fine, if that syntax is internally consistent etc. — Regex are an example of that. There are definitely that could be improved about it, but in itself having a domain-specific language to express pattern matching is a good thing.

(I agree that “go to definition” not working for operators exacerbates this; I’m actually baffled that [AFAIK] no IDE supports this. Don’t Python language servers offer it? Is it a limitation of the language server protocol?)

[-]

somebodddy@reddit

I love Kotlin's approach to that. Kotlin has an officially defines concept of "DSL" which is basically a function that calls a lambda with scope binding to an helper object. While not providing the full expressive power of operator-overloading based DSLs, it's enough for most DSL needs and it's several orders of magnitude simpler. Making it an officially endorsed pattern also makes it easier to understand when encountered in the wild.

[-]

kwargs_@reddit (OP)

Yea. I wonder if that’s a general trade off with syntactic sugar.. makes you feel smart writing it but painful to read/understand. Go has no sugar, it’s boring to write, and always really easy to read.

[-]

guepier@reddit

I don’t know how it happened, but Reddit completely fucked up your markup.

[-]

Dustin-@reddit

result = await ( Pipeline(range(100)) >> process_data >> validate_data

).run()

What do the >> symbols do? I've never seen this syntax before.

[-]

MediumRay@reddit

Seems like he’s taken inspiration from the cpp streaming operator

[-]

kwargs_@reddit (OP)

Typically it’s a bitwise operator right-shift but python is crazy and lets you override the behavior of operators with special class methods.. so in this library it’s a just alias for the stage function.

[-]

light24bulbs@reddit

Concurrency is the best part of go. There are other parts of the language that are completely halfbaked or downright missing. The pointer handling safety and typing (actually the type system in general) is complete ass. But the concurrency, wow. It really works and it's really ergonomic. MUCH better than async await.

[-]

Ancillas@reddit

I don’t know what “ergonomic” means in the context of programming languages.

[-]

kwargs_@reddit (OP)

When I say ergonomic, I mean expressive, convenient, and productive rather than verbose, tedious, clunky. For example, I’ve been following some of the latest developments to Java. The language is becoming more expressive and elegant. Modern Java seems more ergonomic than legacy Java.

[-]

Ancillas@reddit

Okay. Those seem subjective and impossible to measure, but I appreciate it’s something you care about and that you’d take the time to share.

Since you asked for feedback on whether or not people want concurrency to be ergonomic, I’ll offer my opinion. I want APIs that are easy to use while interacting with the hardware in a way that is performant and not wasteful. I would prefer I higher learning curve up front over a library or framework that is less performant. I like rather train my team on the underlying technology than add another layer of abstraction over it to be more immediately accessible. There are always nuances to this, but elegance and expressiveness are not words I’d use when describing the acceptance criteria for selecting a technology choice.

[-]

Rainbow_Plague@reddit

Sounds like you do very different development than me then. I absolutely factor in the "feel" or ease-of-use of libraries and frameworks I use because dev time = money and we'll be working with it every day. Bonus points if it's really easy for new devs to pick up.

[-]

Ancillas@reddit

That makes sense. I find that I lose more money in wasted compute resources than I do training software engineers, so that’s where I optimize my savings.

[-]

PlayfulRemote9@reddit

cool, i'm actually using python for a project where go feels natural cause of concurrency. using this, how would I do a fan out in python? because of the GIL, you can't really do much concurrency

[-]

The_real_bandito@reddit

You could do a hilarious thing and do the same for nodejs using workers

[-]

kwargs_@reddit (OP)

it did cross my mind 😂

[-]

somebodddy@reddit

Curious what people think: do we actually want concurrency to be ergonomic, or is some friction a necessary guardrail?

I'm not fond of this approach. If you have some specific guardrail that can prevent misuse of concurrency (or any other feature, for that matter) it may be worthwhile to implement it even at the cost of reducing ergonomics, but for the sake of creating friction? If you want to deter people from doing concurrency, why even write a concurrency library?

[-]

kwargs_@reddit (OP)

guardrails to make concurrency safer at the cost of ergonomics, yes. deter people from doing concurrency, definitely not what I want. I want more, easier, safer concurrency.