Analyzing language extension semantics

Hi I'm Jappie and I volunteer for the Haskell Foundation Stability Working Group. Recently we analyzed the head.hackage patches to understand why code breaks on new GHC releases. "head.hackage" is a repository of patches for Hackage. GHC engineers use these to test out new GHC builds on a wide range of Hackage packages without having to upstream¹ a patch, which can take time. Instead, they can put the patch in "head.hackage" and immediately test it on a wide range of packages. Surprisingly, most breakage wasn’t caused by Template Haskell, it came from deeper semantic changes in language extensions. The meaning of (some) language extensions changed between GHC releases. This post walks through the main categories of breakage, why they happened, and what they tell us about long-term stability. If you care about a smoother upgrade path for Haskell users, we invite you to participate in the Haskell Foundation Stability Working Group.

Extending our initial investigation, We're also interested in understanding why breakage occurs. So we've recently done further analysis on head.hackage, and learned surprisingly enough that the root cause of a lot of breakage isn't Template Haskell, but seems to be from language extension semantics². We're doing this investigation to understand better where efforts should be focused in improving stability.

This gave us the following table:

th-compat-0.1.4.patch was miscounted so I left that out. Simplified subsumption appears a lot but 3 are for Cabal, so it's only 2 real occurrences. I'd expect that to appear a lot however, because it was one of the motivating changes for a stability working group.

Simplified subsumption

For the blissfully ignorant reader simplified subsumption causes you to do this under certain existential conditions:

--- a/Distribution/Simple/Utils.hs
+++ b/Distribution/Simple/Utils.hs
@@ -1338,7 +1338,7 @@ withTempFileEx opts tmpDir template action =
     (\(name, handle) -> do hClose handle
                            unless (optKeepTempFiles opts) $
                              handleDoesNotExist () . removeFile $ name)
-    (withLexicalCallStack (uncurry action))
+    (withLexicalCallStack (\x -> uncurry action x))

You have to insert a lambda, which apparently has some performance impact. This had a big impact on Yesod stacks, whose code generation helpfully created the database alias in the template:

type DB a = forall (m :: Type -> Type).
    (MonadUnliftIO m) => ReaderT SqlBackend m a

Normally this is quite convenient, however with the simplified subsumption change, any code that interacts with the database now has to insert those lambdas. As you can imagine this would in many places for a commercial code base. Causing a lot of compile errors for industrial users. Instead of inserting lambdas, you can also delete those existential aliases to solve the problem. Or you can enable the language extension: DeepSubsumption. Which restores the original behavior.

Moving of instances due to Template Haskell

This change forces you to put the instances above the splice where they are being used in the same module. A dear colleague decided to generate instances in Template Haskell. That was quite the puzzle! I asked the GHC devs why they did this, and it turns out this was a soundness issue in the typechecker. Here, soundness means the type system can't be tricked into allowing invalid programs. So the community is better off, despite this causing a fair bit of work.

--- a/src/Ipe/Content.hs
+++ b/src/Ipe/Content.hs
@@ -288,6 +288,14 @@

+instance Fractional r => IsTransformable (IpeObject r) where
+  transformBy t (IpeGroup i)     = IpeGroup     $ i&core %~ transformBy t
+ ...
 makePrisms ''IpeObject

@@ -303,14 +311,6 @@

-instance Fractional r => IsTransformable (IpeObject r) where
-  transformBy t (IpeGroup i)     = IpeGroup     $ i&core %~ transformBy t
- ...

(1) Parser change

The parser is the component of the compiler that transforms text into a memory structure the compiler can work with. This structure is called an abstract syntax tree.

-      Node4 {getMBB :: {-# UNPACK #-} ! MBB, getC1 :: ! (RTree a), getC2 :: ! (RTree a), getC3 :: ! (RTree a), getC4 :: ! (RTree a) }
-    | Node3 {getMBB :: {-# UNPACK #-} ! MBB, getC1 :: ! (RTree a), getC2 :: ! (RTree a), getC3 :: ! (RTree a) }
-    | Node2 {getMBB :: {-# UNPACK #-} ! MBB, getC1 :: ! (RTree a), getC2 :: ! (RTree a) }
+      Node4 {getMBB :: {-# UNPACK #-} !MBB, getC1 :: !(RTree a), getC2 :: !(RTree a), getC3 :: !(RTree a), getC4 :: !(RTree a) }
+    | Node3 {getMBB :: {-# UNPACK #-} !MBB, getC1 :: !(RTree a), getC2 :: !(RTree a), getC3 :: !(RTree a) }
+    | Node2 {getMBB :: {-# UNPACK #-} !MBB, getC1 :: !(RTree a), getC2 :: !(RTree a) }
     | Node  {getMBB ::                  MBB, getChildren' :: [RTree a] }
-    | Leaf  {getMBB :: {-# UNPACK #-} ! MBB, getElem :: a}
+    | Leaf  {getMBB :: {-# UNPACK #-} !MBB, getElem :: a}
     | Empty

This is a change from all the way back in 2020, where the core language changed by disallowing ! before parens. Here the bang ! indicates strict fields. Technically this doesn't fit into the category because the core language isn't a language extension. But semantics did change! Actually I don't think we expected to find something like this at all. I'm not sure how relevant this is to discuss further because it appears quite rare for someone to do this. You can enable StrictData in your Cabal file and delete all those bangs!

(2) Rename forall identifiers

This changes the forall identifier into a keyword at term level. It already was at the type level. The issue is discussed here

 hintExplicitForall :: Located Token -> P ()
 hintExplicitForall tok = do
-    forall   <- getBit ExplicitForallBit
+    forAll   <- getBit ExplicitForallBit
     rulePrag <- getBit InRulePragBit
-    unless (forall || rulePrag) $ addError $ mkPlainErrorMsgEnvelope (getLoc tok) $
+    unless (forAll || rulePrag) $ addError $ mkPlainErrorMsgEnvelope (getLoc tok) $
       (PsErrExplicitForall (isUnicode tok))

(3) TypeAbstractions

From what I understand from the manual is that part of the syntax for type abstractions landed in GHC 9.2, however 9.8 and onwards requires you to enable this language extension. This appeared because certain new functionality was introduced behind an old language extension flag, according to this proposal. It says we don't want to introduce new functionality behind established extensions, so that's why we require TypeAbstractions now, where previously ScopedTypeVariables and TypeApplications were enough.

This extension enables you to bind type variables in pattern matches. I don't know why this happened like this, but it happened in 2023:

+-- Invisible type binders in type declarations, such as
+--
+--   type family Sing @k
+--
+-- require the TypeAbstractions extension.
+#if __GLASGOW_HASKELL__ >= 909
+{-# LANGUAGE TypeAbstractions #-}
+#endif
+

(4) Star is type

This change was announced via a warning. It tells users to write Type instead of * for kinds representing types. A kind is essentially the type of a type, and as a concept is used for type-level programming type safety.

-  type Basis v :: *
+  type Basis v :: Type

Conclusion

Often these breakages are annoying and frustrating. But if we look deeper, we find that each of them has a little story and good reasons for being introduced. If you find this all as interesting as I do, please consider joining some of the stability working group meetings!