01 May 2024

feedPlanet Mozilla

Dave Townsend: Tests don't replace Code Review

Featured image of post Tests don't replace Code Review

I frequently see a bold claim come up in tech circles. That as a team you're wasting time by doing code reviews. You should instead rely on automated tests to catch bugs. This surprises me because I can't imagine anyone thinking that such a blanket statement is true. But then most of the time this is brought up in places like Twitter where nuance is impossible and engagement farming is rife. Still it got me thinking about why I think code review is important even if you have amazing tests.

Before I elaborate I'll point out what should be obvious. Different projects have different needs. You shouldn't listen to me tell you that you must do code review any more than you should listen to anyone else tell you that you must not do code review. Be pragmatic in all things. Beware one-size-fits-all statements (in almost any context).

We've been religiously performing code review on every (well almost every) patch at Mozilla since well before I joined the project which was quite some time ago. And in that time I've seen Firefox go from having practically no automated tests to a set of automated test suites that if run end to end on a single machine (which is impossible but let's ignore that) would take nearly two months (😱) to complete. And in that time I don't think I've ever heard anyone suggest we should stop doing code review for anything that actually ships to users (we do allow documentation changes with no review). Why?

# A good set of automated tests doesn't just magically appear

Let's start with the obvious.

Someone has to have written all of those tests. And others have to have verified all that. And even if your test suite is already perfect, how do you know that the developer building a new feature has also included the tests necessary to verify that feature going forwards?

There are some helpful tools that exist, like code coverage. But these are more informative than indicative. Useful to track but should rarely be used by themselves.

# Garbage unmaintainable code passes tests

There are usually many ways to fix a bug or implement a feature. Some of those will be clear readable code with appropriate comments that a random developer in three years time can look at and understand quickly. Others will be spaghetti code that is to all intents and purposes obfuscated. Got a bug in there? It may take ten times longer to fix it. Lint rules can help with this to some extent, but a human code reviewer is going to spot unreadable code a mile away.

# You cannot test everything

It's often not feasible to test for every possible case. Anywhere your code interacts with anything outside of itself, like a filesystem or a network, is going to have cases that are really hard to simulate. What if memory runs out at a critical moment? What if the OS suddenly decides that the disk is not writable? These are cases we have to handle all the time in Firefox. You could say we should build abstractions around everything so that tests can simulate all those cases. But abstractions are not cheap and performance is pretty critical for us.

# But I'm a 100x developer, none of this applies to me

I don't care how senior a developer you are, you'll make mistakes. I sure do. Now it's true that there is something to be said for adjusting your review approach based on the developer who wrote the code. If I'm reviewing a patch by a junior developer I'm going to go over the patch with a fine tooth-comb and then when they re-submit I'm going to take care to make sure they addressed all my changes. Less so with a senior developer who I know knows the code at hand.

# So do tests help with code review at all?

Absolutely!

Tests are there to automatically spot problems, ideally before a change even reaches the review stage. Code review is there to fill in the gaps. You can mostly skip over worrying about whether this breaks well tested functionality (just don't assume all functionality is well tested!). Instead you can focus on what the change is doing that cannot be tested:

Code review and automated tests are complimentary. I believe you'll get the best result when you employ both sensibly. Assuming you have the resources to do so of course. I don't think large projects can do without both.

01 May 2024 10:02pm GMT

Mozilla Thunderbird: Thunderbird Monthly Development Digest: April 2024

Graphic with text "Thunderbird Development Digest April 2024," featuring abstract ASCII art on a dark Thunderbird logo background.

Auf Deutsch übersetzen
Traduire en français
日本語に翻訳

Hello Thunderbird Community, and welcome back to the monthly Thunderbird development digest. April just ended and we're running at full speed into May. We're only a couple of months away from the next ESR, so things are landing faster and we're seeing the finalization of a lot of parallel efforts.

20-Year-Old bugs

Something that has been requested for almost 20 years finally landed on Daily. The ability to control the display of recipients in the message list and better distinguish unknown addresses from those saved in the Address Book was finally implemented in Bug 243258 - Show email address in message list.

This is one of the many examples of features that in the past were very complicated and tricky to implement, but that we were finally able to address thanks to the improvements of our architecture and being able to work with a more flexible and modular code.

We're aiming at going through those very very old requests and slowly addressing them when possible.

Exchange alpha

More Exchange support improvements and features are landing on Daily almost…daily (pun intended). If you want to test things with a local build, you can follow this overview from Ikey.

We will soon look at the possibility of enabling Rust builds by default, making sure that all users will be able to consume our Rust code from next beta, and only needing to switch a pref in order to test Exchange.

Folder compaction

If you've been tracking our most recent struggles, you're probably aware of one of the lingering annoying issues which sees the bubbling up of the size of the user profile caused by local folder corruption.

Ben dive bombed into the code and found a spaghetti mess that was hard to untangle. You can read more about his exploration and discoveries in his recent post on TB-Planning.

We're aiming to land this code hopefully before the end of the week and start calling for some testing and feedback from the community to ensure that all the various issues have been addressed correctly.

You can follow the progress in Bug 1890448 - Rewrite folder compaction.

Cards View

If you're running Beta or Daily, you might have noticed some very fancy new UI for the Cards View. This has been a culmination of many weeks of UX analysis to ensure a flexible and consistent hover, selection, and focus state.

Micah and Sol identified a total of 27 different interaction states on that list, and implementing visual consistency while guaranteeing optimal accessibility levels for all operating systems and potential custom themes was not easy.

We're very curious to hear your feedback.

Context menu

A more refined and updated context menu for the message list also landed on Daily.

A very detailed UX exploration and overview of the implementation was shared on the UX Mailing list a while ago.

This update is only the first step of many more to come, so we apologize in advance if some things are not super polished or things seem temporarily off.

ESR Preview

If you're curious about what the next ESR will look like or checking new features, please consider downloading and installing Beta (preferably in another directory to not override your current profile.) Help us test this new upcoming release and find bugs early.

As usual, if you want to see things as they land you can always check the pushlog and try running daily, which would be immensely helpful for catching bugs early.

See ya next month.

Alessandro Castellani (he, him)
Director, Desktop and Mobile Apps

If you're interested in joining the technical discussion around Thunderbird development, consider joining one or several of our mailing list groups here.

The post Thunderbird Monthly Development Digest: April 2024 appeared first on The Thunderbird Blog.

01 May 2024 4:30pm GMT

This Week In Rust: This Week in Rust 545

Hello and welcome to another issue of This Week in Rust! Rust is a programming language empowering everyone to build reliable and efficient software. This is a weekly summary of its progress and community. Want something mentioned? Tag us at @ThisWeekInRust on Twitter or @ThisWeekinRust on mastodon.social, or send us a pull request. Want to get involved? We love contributions.

This Week in Rust is openly developed on GitHub and archives can be viewed at this-week-in-rust.org. If you find any errors in this week's issue, please submit a PR.

Updates from Rust Community

Newsletters
Project/Tooling Updates
Observations/Thoughts
Rust Walkthroughs
Research
Miscellaneous

Crate of the Week

This week's crate is efs, a no-std ext2 filesystem implementation with plans to add other file systems in the future.

Another week completely devoid of suggestions, but llogiq stays hopeful he won't have to dig for next week's crate all by himself.

Please submit your suggestions and votes for next week!

Call for Testing

An important step for RFC implementation is for people to experiment with the implementation and give feedback, especially before stabilization. The following RFCs would benefit from user testing before moving forward:

If you are a feature implementer and would like your RFC to appear on the above list, add the new call-for-testing label to your RFC along with a comment providing testing instructions and/or guidance on which aspect(s) of the feature need testing.

Call for Participation; projects and speakers

CFP - Projects

Always wanted to contribute to open-source projects but did not know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started!

Some of these tasks may also have mentors available, visit the task page for more information.

If you are a Rust project owner and are looking for contributors, please submit tasks here.

CFP - Speakers

Are you a new or experienced speaker looking for a place to share something cool? This section highlights events that are being planned and are accepting submissions to join their event as a speaker.

If you are an event organizer hoping to expand the reach of your event, please submit a link to the submission website through a PR to TWiR.

Updates from the Rust Project

409 pull requests were merged in the last week

Rust Compiler Performance Triage

Several non-noise changes this week, with both improvements and regresions coming as a result. Overall compiler performance is roughly neutral across the week.

Triage done by @simulacrum. Revision range: a77f76e2..c65b2dc9

2 Regressions, 2 Improvements, 3 Mixed; 1 of them in rollups 51 artifact comparisons made in total

Full report here

Approved RFCs

Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week:

Final Comment Period

Every week, the team announces the 'final comment period' for RFCs and key PRs which are reaching a decision. Express your opinions now.

RFCs
Tracking Issues & PRs

Rust

New and Updated RFCs

Upcoming Events

Rusty Events between 2024-05-01 - 2024-05-29 🦀

Virtual
Africa
Asia
Europe
North America
Oceania

If you are running a Rust event please add it to the calendar to get it mentioned here. Please remember to add a link to the event too. Email the Rust Community Team for access.

Jobs

Please see the latest Who's Hiring thread on r/rust

Quote of the Week

"I'll never!" "No, never is in the 2024 Edition." "But never can't be this year, it's never!" "Well we're trying to make it happen now!" "But never isn't now?" "I mean technically, now never is the unit." "But how do you have an entire unit if it never happens?"

- Jubilee on Zulip

Thanks to Jacob Pratt for the suggestion!

Please submit quotes and vote for next week!

This Week in Rust is edited by: nellshamrell, llogiq, cdmistman, ericseppanen, extrawurst, andrewpollack, U007D, kolharsam, joelmarcey, mariannegoldin, bennyvasquez.

Email list hosting is sponsored by The Rust Foundation

Discuss on r/rust

01 May 2024 4:00am GMT

William Durand: Moziversary #6

Today is my sixth Moziversary 🎂 I joined Mozilla as a full-time employee on May 1st, 2018. I previously blogged in 2019, 2020, 2021, 2022, and 2023.

Last year, I mainly contributed to Firefox for Android as the lead engineer on a project called "Add-ons General Availability (GA)". The goal was to allow for more add-ons on this platform. Success! More than a thousand extensions are now available on Android 🎉

In addition, I worked on a Firefox feature called Quarantined Domains and implemented a new abuse report form on addons.mozilla.org (AMO) to comply with the Digital Services Act (DSA). I was also involved in two other cross-team efforts related to the Firefox installation funnel. I investigated various issues (e.g. this openSUSE bug), and I coordinated the deprecation of weak add-on signatures and some more changes around certificates lately, which is why I wrote xpidump.

Phew! There is no shortage of work.

When I moved to the WebExtensions team in 2022, I wrote about this incredible challenge. I echoed this sentiment several months later in my 2022 Moziversary update. I couldn't imagine how much I would achieve in two years…

Back then, I didn't know what the next step in my career would be. I have been aiming to bridge the gap between the AMO and WebExtensions engineering teams since at least 2021 and that is my "next step".

I recently took a new role as Add-ons Tech Lead. This is the continuation of what I've been doing for some time but that comes with new challenges and opportunities as well. We'll see how it goes but I am excited!

I'll be forever grateful to my manager and coworkers. Thank you ❤️

01 May 2024 12:00am GMT

The Rust Programming Language Blog: Announcing Google Summer of Code 2024 selected projects

The Rust Project is participating in Google Summer of Code (GSoC) 2024, a global program organized by Google which is designed to bring new contributors to the world of open-source.

In February, we published a list of GSoC project ideas, and started discussing these projects with potential GSoC applicants on our Zulip. We were pleasantly surprised by the amount of people that wanted to participate in these projects and that led to many fruitful discussions with members of various Rust teams. Some of them even immediately began contributing to various repositories of the Rust Project, even before GSoC officially started!

After the initial discussions, GSoC applicants prepared and submitted their project proposals. We received 65 (!) proposals in total. We are happy to see that there was so much interest, given that this is the first time the Rust Project is participating in GSoC.

A team of mentors primarily composed of Rust Project contributors then thoroughly examined the submitted proposals. GSoC required us to produce a ranked list of the best proposals, which was a challenging task in itself since Rust is a big project with many priorities! We went through many rounds of discussions and had to consider many factors, such as prior conversations with the given applicant, the quality and scope of their proposal, the importance of the proposed project for the Rust Project and its wider community, but also the availability of mentors, who are often volunteers and thus have limited time available for mentoring.

In many cases, we had multiple proposals that aimed to accomplish the same goal. Therefore, we had to pick only one per project topic despite receiving several high-quality proposals from people we'd love to work with. We also often had to choose between great proposals targeting different work within the same Rust component to avoid overloading a single mentor with multiple projects.

In the end, we narrowed the list down to twelve best proposals, which we felt was the maximum amount that we could realistically support with our available mentor pool. We submitted this list and eagerly awaited how many of these twelve proposals would be accepted into GSoC.

Selected projects

On the 1st of May, Google has announced the accepted projects. We are happy to announce that 9 proposals out of the twelve that we have submitted were accepted by Google, and will thus participate in Google Summer of Code 2024! Below you can find the list of accepted proposals (in alphabetical order), along with the names of their authors and the assigned mentor(s):

Congratulations to all applicants whose project was selected! The mentors are looking forward to working with you on these exciting projects to improve the Rust ecosystem. You can expect to hear from us soon, so that we can start coordinating the work on your GSoC projects.

We would also like to thank all the applicants whose proposal was sadly not accepted, for their interactions with the Rust community and contributions to various Rust projects. There were some great proposals that did not make the cut, in large part because of limited review capacity. However, even if your proposal was not accepted, we would be happy if you would consider contributing to the projects that got you interested, even outside GSoC! Our project idea list is still actual, and could serve as a general entry point for contributors that would like to work on projects that would help the Rust Project maintainers and the Rust ecosystem.

Assuming our involvement in GSoC 2024 is successful, there's a good chance we'll participate next year as well (though we can't promise anything yet) and we hope to receive your proposals again in the future! We also are planning to participate in similar programs in the very near future. Those announcements will come in separate blog posts, so make sure to subscribe to this blog so that you don't miss anything.

The accepted GSoC projects will run for several months. After GSoC 2024 finishes (in autumn of 2024), we plan to publish a blog post in which we will summarize the outcome of the accepted projects.

01 May 2024 12:00am GMT

Don Marti: blog fix: remove stray files

Another update from the blog. Quick recap: I'm re-doing this blog with mostly Pandoc and make, with a few helper scripts.

This is a personal web site and can be broken sometimes, and one of the breakage problems was: oops, I removed a draft post from the directory of source files (in CommonMark) but the HTML version got built and put in public and copied to the server, possibly also affecting the index.html and the RSS feed.

If you're reading the RSS and got some half-baked drafts, that's why.

So, to fix it, I need to ask make if there's anything in the public directory that doesn't have a corresponding source file or files and remove it. Quick helper script:

That should mean a better RSS reading experience since you shouldn't get it cluttered up with drafts if I make a mistake.

But I'm sure I have plenty of other mistakes I can make.

Related

planning for SCALE 2025

Automatically run make when a file changes

Bonus links

We can have a different web Nothing about the web has changed that prevents us from going back. If anything, it's become a lot easier.

A Lawsuit Argues Meta Is Required by Law to Let You Control Your Own Feed (Section 230 protection for a research extension? Makes sense to me.)

Effects of Banning Targeted Advertising (The top 10 percent of Android apps for kids did better after an ad personalization policy change, while the bottom 90 percent lost revenue. If Sturgeon's Law applies to Android apps, the average under-13 user might be better off?)

In Response To Google (Does anyone else notice more and more people working on ways to fix their personal information environment because of the search quality crisis? This blog series from Ed Zitron has some good background.)

01 May 2024 12:00am GMT

30 Apr 2024

feedPlanet Mozilla

Support.Mozilla.Org: What’s up with SUMO — Q1 2024

Hi everybody,

It's always exciting to start a new year as it provides renewed spirit. Even more exciting because the CX team welcomed a few additional members this quarter, including Konstantina, who will be with us crafting better community experiences in SUMO. This is huge, since the SUMO community team has been under resourced for the past few years. I'm personally super excited about this. There are a few things that we're working on internally, and I can't wait to share them with you all. But first thing first, let's read the recap of what happened and what we did in Q1 2024!

Welcome note and shout-outs

If you know someone who you'd like to feature here, please contact Kiki, and we'll make sure to add them in our next edition.

Community news

Stay updated

Community stats

KB

KB pageviews

Month Page views Vs previous month
Jan 2024 6,743,722 3.20%
Feb 2024 7,052,665 4.58%
Mar 2024 6,532,175 -7.38% 
KB pageviews number is a total of English (en-US) KB pageviews

Top 5 KB contributors in the last 90 days:

KB Localization

Top 10 locales based on total page views

Locale/pageviews

Jan 2024

 Feb 2024

Mar 2024

Localization progress (per Apr, 23)
de 2,425,154 2,601,865 2,315,952 92%
fr 1,559,222 1,704,271 1,529,981 81%
zh-CN 1,351,729 1,224,284 1,306,699 100%
es 1,171,981 1,353,200 1,212,666 25%
ja 1,019,806 1,068,034 1,051,625 34%
ru 801,370 886,163 812,882 100%
pt-BR 661,612 748,185 714,554 42%
zh-TW 598,085 623,218 366,320 3%
It 533,071 575,245 529,887 96%
pl 489,532 532,506 454,347 84% 
Locale pageviews is an overall pageview from the given locale (KB and other pages)

Localization progress is the percentage of localized articles from all KB articles per locale

Top 5 localization contributors in the last 90 days:

Forum Support

Forum stats

Month Total questions Answer rate within 72 hrs Solved rate within 72 hrs Forum helpfulness
Jan 2024 2999 72.6% 10.8% 61.3%
Feb 2024 2766 72.4% 9.5% 65.6%
Mar 2024 2516 71.5% 10.4% 71.6%

Top 5 forum contributors in the last 90 days:

Social Support

Month Total replies Total interactions Respond conversion rate
Jan 2024 33 46 71.74%
Feb 2024 25 65 38.46%
Mar 2024 14 87 16.09%

Top 5 Social Support contributors in the past 3 months:

Play Store Support

Month Total replies Total interactions Respond conversion rate
Jan 2024 76 276 27.54%
Feb 2024 49 86 56.98%
Mar 2024 47 80 58.75%

Top 5 Play Store contributors in the past 3 months:

Stay connected

30 Apr 2024 4:15pm GMT

27 Apr 2024

feedPlanet Mozilla

Mozilla Privacy Blog: The UK’s Digital Markets, Competition, and Consumers Bill will spark the UK’s digital economy, not stifle it

In today's digital age, an open and competitive ecosystem with a diverse range of players is essential for building a resilient economy. New products and ideas must have the opportunity to grow to give people meaningful choices. Yet, this reality often falls short due to the dominance of a handful of large companies that create walled gardens by self-preferencing their services over independent competitors - limiting choice and hampering innovation.

The UK's Digital Markets, Competition, and Consumers Bill (DMCCB) offers a unique opportunity to break down these barriers, paving the way for a more competitive and consumer-centric digital market. On the competition side, the DMCCB offers flexibility in allowing for targeted codes of conduct to regulate the behaviour of dominant players. This agile and future-proof approach makes it unique in the ex-ante interventions being considered around the world to rein in abuse in digital markets. An example of what such a code of conduct might look like in practice is the voluntary commitments given by Google to the CMA in the Privacy Sandbox case.

Mozilla, in line with our long history of supporting pro-competition regulatory interventions, supports the DMCCB and its underlying goal of fostering competition by empowering consumers. However, to truly deliver on this promise, the law must be robust, effective, and free from loopholes that could undermine its intent.

Last month, the House of Lords made some much needed improvements to the DMCCB - which are now slated to be debated in the House of Commons in late April/early May 2024. A high-level overview of the key positive changes and why they should remain a part of the law are:

Mozilla remains committed to supporting the DMCCB's swift passage through Parliament and ensuring that it delivers on its promise to empower consumers and promote innovation. We launched a petition earlier today to help push the law over the finish line. By addressing the key concerns we've highlighted above and maintaining a robust framework, the UK can set a global standard for digital markets and create an environment where consumers are truly in charge.

The post The UK's Digital Markets, Competition, and Consumers Bill will spark the UK's digital economy, not stifle it appeared first on Open Policy & Advocacy.

27 Apr 2024 5:17am GMT

Don Marti: realistically get rid of third-party cookies

How would a browser realistically get rid of third-party cookies, if the plan was to just replace third-party cookies, and the project requirements did not include a bunch of anticompetitive tricks too?

  1. Start offering a very scary dialog to a fraction of new users. Something like Do you want to test a new experimental feature? It might-maybe-have some privacy benefits but many sites will break. Don't expect a lot of people to agree at first.

  2. Turn off third-party cookies for the users who did say yes in step 1, and watch the telemetry. There will be positive and negative effects, but they won't be overwhelmingly bad because most sites have to work with other browsers.

  3. When the breakage detected in step 2 gets to be insignificant as a cause of new browser users quitting or reinstalling, start making the dialog less scary and show it to more people.

  4. Keep repeating until most new installs are third-party cookie-free, then start offering the dialog on browser upgrades.

  5. Continue, for more and more users, until you get to 95-99%. Leave the third-party cookies on for 1-5% of users for a couple of releases just to spot any lingering problems, then make third-party cookies default off, with no dialog (users would have to find the preference to re-enable them, or their sysadmin would have to push out a centralized change if some legacy corporate site still needs them).

But what about the personalized ads? Some people actually want those! Not a problem. The good news is that ad personalization can be done in an extension. Ask extension developers who have extensions that support ad personalization to sign up for a registry of ad personalization extensions, then keep track of how many users are installing each one. Adtech firms don't (usually?) have personalization extensions today, but every company can develop one on its own schedule, with less uncertainty and fewer dependencies and delays than the current end of cookies mess. The extension development tools are really good now.

As soon as an ad personalization extension can pass an independent security audit (done by a company agreed on by the extension developer and the browser vendor) and get, say, 10,000 users, then the browser can put it on a choice screen that gets shown for new installs and, if added since last upgrade, upgrades. (The browser could give the dogmatic anti-personalization users a preference to opt out of these choice screens if they really wanted to dig in and find it.) This makes the work of competition regulators much easier-they just have to check that the browser vendor's own ad personalization extension gets fair treatment with competing ones.

And we're done. The privacy people and the personalized ad people get what they want with much less drama and delay, the whole web ad business isn't stuck queued up waiting for one development team, and all that's missing is the anticompetitive stuff that has been making end of cookies work such a pain since 2019.

Related

the 30-40-30 rule An updated list of citations to user research on how many people want personalized ads

Can database marketing sell itself to the people in the database? Some issues that an ad personalization extension might have to address in order to get installs

User tracking as Chesterton's Fence What tracking-based advertising still offers (that alternatives don't)

Catching up to Safari? Some features that Apple has done right, with opportunities for other browsers to think different(ly)

Bonus links

An open letter to the advertising punditry I personally got involved in the Inventory Quality reviews to make sure that the data scientists weren't pressured by the business and could find the patterns-like ww3 [dot] forbes [dot] com-and go after them.

The Rise of Large-Language-Model Optimization The advent of AI threatens to destroy the complex online ecosystem that allows writers, artists, and other creators to reach human audiences.

The Man Who Killed Google Search [M]any found that the update mostly rolled back changes, and traffic was increasing to sites that had previously been suppressed by Google Search's "Penguin" update from 2012 that specifically targeted spammy search results, as well as those hit by an update from an August 1, 2018…

Lawsuit in London to allege Grindr shared users' HIV status with ad firms (This is why you can safely mute anybody who uses the expression k-anonymity, the info about yourself that you most want to keep private is true for more than k other people.)

UK children bombarded by gambling ads and images online, charity warns (attention parents: copy the device rules that Big Tech big shots maintain for their own children, not what they want for yours)

27 Apr 2024 12:00am GMT

26 Apr 2024

feedPlanet Mozilla

Mozilla Privacy Blog: Work Gets Underway on a New Federal Privacy Proposal

At Mozilla, safeguarding privacy has been core to our mission for decades - we believe that individuals' security and privacy on the Internet are fundamental and must not be treated as optional. We have long advocated for a federal privacy law to ensure consumers have control over their data and that companies are accountable for their privacy practices.

Earlier this month, House Committee on Energy and Commerce Chair Cathy McMorris Rodgers (R-WA) and Senate Committee on Commerce, Science and Transportation Chair Maria Cantwell (D-WA) unveiled a discussion draft of the American Privacy Rights Act of 2024 (APRA). The Act is a welcome bipartisan effort to create a unified privacy standard across the United States, with the promise of finally protecting the privacy of all Americans.

At Mozilla, we are committed to the principle of data minimization - a concept that's fundamental in effective privacy legislation - and we are pleased to see it at the core of APRA. Data minimization means we conscientiously collect only the necessary data, ensure its protection, and provide clear and concise explanations about what data we collect and why. We are also happy to see additional strong language from the American Data Privacy and Protect Act (ADPPA) reflected in this new draft, including non-discrimination provisions and a universal opt-out mechanism (though we support clarification that ensures allowance of multiple mechanisms).

However, the APRA discussion draft has open questions that must be refined. These include how APRA handles protections for children, options for strengthening data brokers provisions even further (such as a centralized mechanism for opt-out rights), and key definitions that require clarity around advertising. We look forward to engaging with policymakers as the process advances.

Achieving meaningful reform in the U.S. is long overdue. In an era where digital privacy concerns are on the rise, it's essential to establish clear and enforceable privacy rights for all Americans. Mozilla stands ready to contribute to the dialogue on APRA and collaborate toward achieving comprehensive privacy reform. Together, we can prioritize the interests of individuals and cultivate trust in the digital ecosystem.

The post Work Gets Underway on a New Federal Privacy Proposal appeared first on Open Policy & Advocacy.

26 Apr 2024 7:48pm GMT

Mozilla Privacy Blog: Net Neutrality is Back!

Yesterday, the Federal Communications Commission (FCC) voted 3-2 to reinstate net neutrality rules and protect consumers online. We applaud this decision to keep the internet open and accessible to all, and reverse the 2018 roll-back of net neutrality protections. Alongside our many partners and allies, Mozilla has been a long time proponent of net neutrality across the world and in U.S. states, and mobilized hundreds of thousands of people over the years.

The new FCC order reclassifies broadband internet as a "telecommunications service" and prevents ISPs from blocking, throttling, or paid prioritization of traffic. This action restores meaningful and enforceable FCC oversight and protection on the internet, and unlocks innovation, competition, and free expression online.

You can read Mozilla's submission to the FCC on the proposed Safeguarding and Securing the Open Internet rules in December 2023 here and additional reply comments in January 2024 here.

Net neutrality and openness are essential parts of how we experience the internet, and as illustrated during the COVID pandemic, can offer important protections - so it shouldn't come as a surprise that such a majority of Americans support it. Yesterday's decision reaffirms the internet is and should remain a public resource, where companies cannot abuse their market power to the detriment of consumers, and where actors large and small operate on a level playing field.

Earlier this month, Mozilla participated in a roundtable discussion with experts and allies hosted by Chairwoman Rosenworcel at the Santa Clara County Fire Department. The event location highlighted the importance of net neutrality, as the site where Verizon throttled firefighters' internet speeds in the midst of fighting a raging wildfire. You can watch the full press conference below, and read coverage of the event here.

We thank the FCC for protecting these vital net neutrality safeguards, and we look forward to seeing the details of the final order when released.

The post Net Neutrality is Back! appeared first on Open Policy & Advocacy.

26 Apr 2024 7:28pm GMT

The Servo Blog: This month in Servo: Acid2 redux, Servo book, Qt demo, and more!

Servo nightly, now rendering Acid2 perfectly <figcaption>Servo now renders Acid2 perfectly, but like all browsers, only at 1x dpi.</figcaption>

Back in November, Servo's new layout engine passed Acid1, and this month, thanks to a bug-squashing sprint by @mrobinson and @Loirooriol, we now pass Acid2!

We would also like to thank you all for your generous support! Since we moved to Open Collective and GitHub Sponsors in March, we have received 1578 USD (after fees), including 1348 USD/month (before fees) in recurring donations. This smashed our first two goals, and is a respectable part of the way towards our next goal of 10000 USD/month. For more details, see our Sponsorship page and announcement post.

1348 USD/month
10000

We are still receiving donations from 19 people on LFX, and we're working on transferring the balance to our new fund, but we will stop accepting donations there soon - please move your recurring donations to GitHub or Open Collective. As always, use of these funds will be decided transparently in the Technical Steering Committee, starting with the TSC meeting on 29 April.

The Servo book, a book much like the Rust book

Servo's docs are moving to the Servo book, and a very early version of this is now online (@delan, servo/book)! The goal is to unify our many sources of documentation, including the hacking quickstart guide, building Servo page, Servo design page, and other in-tree docs and wiki pages, into a book that's richer and easier to search and navigate.

Servo now supports several new features in its nightly builds:

As of 2024-04-05, we now support non-autoplay <video> (@eerii, media#419, #32001), as long as the page provides its own controls, as well as the 'baseline-source' property (@MunishMummadi, #31904, #31913). Both of these contributors started out as Outreachy participants, and we're thrilled to see their continued work on improving Servo.

We've also landed several other rendering improvements:

Our font rendering has improved, with support for selecting the correct weight and style in indexed fonts (.ttc) on Linux (@mukilan, @mrobinson, #32127), as well as support for emoji font fallback on macOS (@mrobinson, #32122). Note that color emoji are not yet supported.

Other big changes are coming to Servo's font loading and rendering, thanks to @mrobinson's font system redesign RFC (#32033). Work has already started on this (@mrobinson, @mukilan, #32034, #32038, #32100, #32101, #32115), with the eventual goal of making font data zero-copy readable from multiple threads. This in turn will fix several major issues with font caching, including cached font data leaking over time and between pages, unnecessary loading from disk, and unnecessary copying to layout.

We've also started simplifying the script-layout interface (@mrobinson, #31937, #32081), since layout was merged into the script thread, and script can now call into layout without IPC.

Embedding and dev changes

Servo running in a Qt app via CXX-Qt <figcaption>The prototype shows that Servo can be integrated with a Qt app via CXX-Qt.</figcaption>

A prototype for integrating Servo with Qt was built by @ahayzen-kdab and @vimpostor and shown at Embedded World 2024. We're looking forward to incorporating their feedback from this to improve Servo's embedding API. For more details, check out their GitHub repo and Embedding the Servo Web Engine in Qt.

Servo now supports multiple concurrent webviews (@wusyong, @delan, @atbrakhi, #31417, #32067)! This is a big step towards making Servo a viable embedded webview, and we will soon use it to implement tabbed browsing in servoshell (@delan, #31545).

Three of the slowest crates in the Servo build process are mozjs_sys, mozangle, and script. The first two compile some very large C++ libraries in their build scripts - SpiderMonkey and ANGLE respectively - and the third blocks on the first two. They can account for over two minutes of build time, even on a very fast machine (AMD 7950X), and a breaking change in newer versions of GNU Make (mozjs#375) can make mozjs_sys take over eight minutes to build!

mozjs_sys now uses a prebuilt version of SpiderMonkey by default (@wusyong, @sagudev, mozjs#450, #31824), cutting clean build times by over seven minutes on a very fast machine (see above). On Linux with Nix (the package manager), where we run an unaffected version of GNU Make, it can still save over 100 seconds on a quad-core CPU with SMT. Further savings will be possible once we do the same for mozangle.

If you use NixOS, or any Linux distro with Nix, you can now get a shell with all of the tools and dependencies needed to build and run Servo by typing nix-shell (@delan, #32035), without also needing to type etc/shell.nix.

As for CI, our experimental Android build now supports aarch64 (@mukilan, #32137), in addition to Android on armv7, x86_64, and i686, and we've improved flakiness in the WebGPU tests (@sagudev, #31952) and macOS builds (@mrobinson, #32005).

Conferences and events

Earlier this month, Rakhi Sharma gave her talk A year of Servo reboot: where are we now? at Open Source Summit North America (slides; recording available soon) and at the Seattle Rust User Group (slides).

In the Netherlands, Gregory Terzian will be presenting Modular Servo: Three Paths Forward at the GOSIM Conference 2024, on 6 May at 15:10 local time (13:10 UTC). That's the same venue as RustNL 2024, just one day earlier, and you can also find Gregory, Rakhi, and Nico at RustNL afterwards. See you there!

26 Apr 2024 12:00am GMT

25 Apr 2024

feedPlanet Mozilla

Will Kahn-Greene: crashstats-tools v2.0.0 released!

What is it?

crashstats-tools is a set of command-line tools for working with Crash Stats (https://crash-stats.mozilla.org/).

crashstats-tools comes with four commands:

  • supersearch: for performing Crash Stats Super Search queries

  • supersearchfacet: for performing aggregations, histograms, and cardinality Crash Stats Super Search queries

  • fetch-data: for fetching raw crash, dumps, and processed crash data for specified crash ids

  • reprocess: for sending crash report reprocess requests

v2.0.0 released!

There have been a lot of improvements since the last blog post for the v1.0.1 release. New commands, new features, improved cli ui, etc.

v2.0.0 focused on two major things:

  1. improving supersearchfacet to support nested aggregation, histogram, and cardinality queries

  2. moving some of the code into a crashstats_tools.libcrashstats module improving its use as a library

Improved supersearchfacet

The other day, Alex and team finished up the crash reporter Rust rewrite. The crash reporter rewrite landed and is available in Firefox, nightly channel, where build_id >= 20240321093532.

The crash reporter is one of the clients that submits crash reports to Socorro which is now maintained by the Observability Team. Firefox has multiple crash reporter clients and there are many ways that crash reports can get submitted to Socorro.

One of the changes we can see in the crash report data now is the change in User-Agent header. The new rewritten crash reporter sends a header of crash-reporter/1.0.0. That gets captured by the collector and put in the raw crash metadata.user_agent field. It doesn't get indexed, so we can't search on it directly.

We can get a sampling of the last 100 crash reports, download the raw crash data, and look at the user agents.

$ supersearch --num=100 --product=Firefox --build_id='>=20240321093532' \
    --release_channel=nightly > crashids.txt
$ fetch-data --raw --no-dumps --no-processed crashdata < crashids.txt
$ jq .metadata.user_agent crashdata/raw_crash/*/* | sort | uniq -c
     16 "crashreporter/1.0.0"
      2 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0"
      1 "Mozilla/5.0 (Windows NT 10.0; rv:127.0) Gecko/20100101 Firefox/127.0"
      2 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"
     63 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0"
      1 "Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"
     12 "Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
      3 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"

16 out of 100 crash reports were submitted by the new crash reporter. We were surprised there are so many Firefox user agents. We discussed this on Slack. I loosely repeat it here because it's a great way to show off some of the changes of supersearchfacet in v2.0.0.

First, the rewritten crash reporter only affects the parent (aka main) process. The other processes have different crash reporters that weren't rewritten.

How many process types are there for Firefox crash reports in the last week? We can see that in the ProcessType annotation (docs) which is processed and saved in the process_type field (docs).

$ supersearchfacet --product=Firefox --build_id='>=20240321093532' --release_channel=nightly
    --_facets=process_type
process_type
 process_type | count
--------------|-------
 content      | 3664
 parent       | 2323
 gpu          | 855
 utility      | 225
 rdd          | 60
 plugin       | 18
 socket       | 2
 total        | 7147

Judging by that output, I would expect to see a higher percentage of crashreporter/1.0.0 in our sampling of 100 crash reports.

Turns out that Firefox uses different code to submit crash reports not just by process type, but also by user action. That's in the SubmittedFrom annotation (docs) which is processed and saved in the submitted_from field (docs).

$ supersearchfacet --product=Firefox --build_id='>=20240321093532' --release_channel=nightly \
    --_facets=submitted_from
submitted_from
 submitted_from | count
----------------|-------
 Auto           | 3477
 Client         | 1741
 CrashedTab     | 928
 Infobar        | 792
 AboutCrashes   | 209
 total          | 7147

What is "Auto"? The user can opt-in to auto-send crash reports. When Firefox upgrades and this setting is set, then Firefox will auto-send any unsubmitted crash reports. The nightly channel has two updates a day, so there's lots of opportunity for this event to trigger.

What're the counts for submitted_from/process_type pairs?

$ supersearchfacet --product=Firefox --build_id='>=20240321093532' --release_channel=nightly \
    --_aggs.process_type=submitted_from
process_type / submitted_from
 process_type / submitted_from | count
-------------------------------|-------
 content / Auto                | 2214
 content / CrashedTab          | 926
 content / Infobar             | 399
 content / AboutCrashes        | 125
 parent / Client               | 1741
 parent / Auto                 | 450
 parent / Infobar              | 107
 parent / AboutCrashes         | 25
 gpu / Auto                    | 565
 gpu / Infobar                 | 236
 gpu / AboutCrashes            | 54
 utility / Auto                | 198
 utility / Infobar             | 25
 utility / AboutCrashes        | 2
 rdd / Auto                    | 34
 rdd / Infobar                 | 23
 rdd / AboutCrashes            | 3
 plugin / Auto                 | 14
 plugin / CrashedTab           | 2
 plugin / Infobar              | 2
 socket / Auto                 | 2
 total                         | 7147

We can spot check these different combinations to see what the user-agent looks like.

For brevity, we'll just look at parent / Client in this blog post.

$ supersearch --num=100 --product=Firefox --build_id='>=20240321093532' --release_channel=nightly \
    --process_type=parent --submitted_from='~Client' > crashids_clarified.txt
$ fetch-data --raw --no-dumps --no-processed crashdata_clarified < crashids_clarified.txt
$ jq .metadata.user_agent crashdata_clarified/raw_crash/*/* | sort | uniq -c
    100 "crashreporter/1.0.0"

Seems like the crash reporter rewrite only affects crash reports where ProcessType=parent and SubmittedFrom=Client. All the other process_type/submitted_from combinations get submitted a different way where the user agent is the browser itself.

How many crash reports has the new crash reporter submitted over time?

$ supersearchfacet --_histogram.date=product --_histogram.interval=1d --denote-weekends \
    --date='>=2024-03-20' --date='<=2024-04-25' \
    --release_channel=nightly --product=Firefox --build_id='>=20240321093532' \
    --submitted_from='~Client' --process_type=parent
histogram_date.product
 histogram_date | Firefox | total
----------------|---------|-------
 2024-03-21     | 58      | 58
 2024-03-22     | 124     | 124
 2024-03-23 **  | 189     | 189
 2024-03-24 **  | 289     | 289
 2024-03-25     | 202     | 202
 2024-03-26     | 164     | 164
 2024-03-27     | 199     | 199
 2024-03-28     | 187     | 187
 2024-03-29     | 188     | 188
 2024-03-30 **  | 155     | 155
 2024-03-31 **  | 146     | 146
 2024-04-01     | 201     | 201
 2024-04-02     | 226     | 226
 2024-04-03     | 236     | 236
 2024-04-04     | 266     | 266
 2024-04-05     | 259     | 259
 2024-04-06 **  | 227     | 227
 2024-04-07 **  | 214     | 214
 2024-04-08     | 259     | 259
 2024-04-09     | 257     | 257
 2024-04-10     | 223     | 223
 2024-04-11     | 250     | 250
 2024-04-12     | 235     | 235
 2024-04-13 **  | 154     | 154
 2024-04-14 **  | 162     | 162
 2024-04-15     | 207     | 207
 2024-04-16     | 201     | 201
 2024-04-17     | 346     | 346
 2024-04-18     | 270     | 270
 2024-04-19     | 221     | 221
 2024-04-20 **  | 190     | 190
 2024-04-21 **  | 183     | 183
 2024-04-22     | 266     | 266
 2024-04-23     | 303     | 303
 2024-04-24     | 308     | 308

There are more examples in the crashstats-tools README.

crashstats_tools.libcrashstats library

Starting with v2.0.0, you can use crashstats_tools.libcrashstats as a library for Python scripts.

For example:

from crashstats_tools.libcrashstats import supersearch

results = supersearch(params={"_columns": "uuid"}, num_results=100)

for result in results:
    print(f"{result}")

libcrashstats makes using the Crash Stats API a little more ergonomic.

See the crashstats_tools.libcrashstats library documentation.

Be thoughtful about using data

Make sure to use these tools in compliance with our data policy:

https://crash-stats.mozilla.org/documentation/protected_data_access/

Where to go for more

See the project on GitHub which includes a README which contains everything about the project including examples of usage, the issue tracker, and the source code:

https://github.com/willkg/crashstats-tools

Let me know whether this helps you!

25 Apr 2024 4:00pm GMT

Hacks.Mozilla.Org: Llamafile’s progress, four months in

When Mozilla's Innovation group first launched the llamafile project late last year, we were thrilled by the immediate positive response from open source AI developers. It's become one of Mozilla's top three most-favorited repositories on GitHub, attracting a number of contributors, some excellent PRs, and a growing community on our Discord server.

Through it all, lead developer and project visionary Justine Tunney has remained hard at work on a wide variety of fundamental improvements to the project. Just last night, Justine shipped the v0.8 release of llamafile, which includes not only support for the very latest open models, but also a number of big performance improvements for CPU inference.

As a result of Justine's work, today llamafile is both the easiest and fastest way to run a wide range of open large language models on your own hardware. See for yourself: with llamafile, you can run Meta's just-released LLaMA 3 model-which rivals the very best models available in its size class-on an everyday Macbook.

How did we do it? To explain that, let's take a step back and tell you about everything that's changed since v0.1.

tinyBLAS: democratizing GPU support for NVIDIA and AMD

llamafile is built atop the now-legendary llama.cpp project. llama.cpp supports GPU-accelerated inference for NVIDIA processors via the cuBLAS linear algebra library, but that requires users to install NVIDIA's CUDA SDK. We felt uncomfortable with that fact, because it conflicts with our project goal of building a fully open-source and transparent AI stack that anyone can run on commodity hardware. And besides, getting CUDA set up correctly can be a bear on some systems. There had to be a better way.

With the community's help (here's looking at you, @ahgamut and @mrdomino!), we created our own solution: it's called tinyBLAS, and it's llamafile's brand-new and highly efficient linear algebra library. tinyBLAS makes NVIDIA acceleration simple and seamless for llamafile users. On Windows, you don't even need to install CUDA at all; all you need is the display driver you've probably already installed.

But tinyBLAS is about more than just NVIDIA: it supports AMD GPUs, as well. This is no small feat. While AMD commands a respectable 20% of today's GPU market, poor software and driver support have historically made them a secondary player in the machine learning space. That's a shame, given that AMD's GPUs offer high performance, are price competitive, and are widely available.

One of llamafile's goals is to democratize access to open source AI technology, and that means getting AMD a seat at the table. That's exactly what we've done: with llamafile's tinyBLAS, you can now easily make full use of your AMD GPU to accelerate local inference. And, as with CUDA, if you're a Windows user you don't even have to install AMD's ROCm SDK.

All of this means that, for many users, llamafile will automatically use your GPU right out of the box, with little to no effort on your part.

CPU performance gains for faster local AI

Here at Mozilla, we are keenly interested in the promise of "local AI," in which AI models and applications run directly on end-user hardware instead of in the cloud. Local AI is exciting because it opens up the possibility of more user control over these systems and greater privacy and security for users.

But many consumer devices lack the high-end GPUs that are often required for inference tasks. llama.cpp has been a game-changer in this regard because it makes local inference both possible and usably performant on CPUs instead of just GPUs.

Justine's recent work on llamafile has now pushed the state of the art even further. As documented in her detailed blog post on the subject, by writing 84 new matrix multiplication kernels she was able to increase llamafile's prompt evaluation performance by an astonishing 10x compared to our previous release. This is a substantial and impactful step forward in the quest to make local AI viable on consumer hardware.

This work is also a great example of our commitment to the open source AI community. After completing this work we immediately submitted a PR to upstream these performance improvements to llama.cpp. This was just the latest of a number of enhancements we've contributed back to llama.cpp, a practice we plan to continue.

Raspberry Pi performance gains

Speaking of consumer hardware, there are few examples that are both more interesting and more humble than the beloved Raspberry Pi. For a bargain basement price, you get a full-featured computer running Linux with plenty of computing power for typical desktop uses. It's an impressive package, but historically it hasn't been considered a viable platform for AI applications.

Not any more. llamafile has now been optimized for the latest model (the Raspberry Pi 5), and the result is that a number of small LLMs-such as Rocket-3B (download), TinyLLaMA-1.5B (download), and Phi-2 (download)-run at usable speeds on one of the least expensive computers available today. We've seen prompt evaluation speeds of up to 80 tokens/sec in some cases!

Keeping up with the latest models

The pace of progress in the open model space has been stunningly fast. Over the past few months, hundreds of models have been released or updated via fine-tuning. Along the way, there has been a clear trend of ever-increasing model performance and ever-smaller model sizes.

The llama.cpp project has been doing an excellent job of keeping up with all of these new models, frequently rolling-out support for new architectures and model features within days of their release.

For our part we've been keeping llamafile closely synced with llama.cpp so that we can support all the same models. Given the complexity of both projects, this has been no small feat, so we're lucky to have Justine on the case.

Today, you can today use the very latest and most capable open models with llamafile thanks to her hard work. For example, we were able to roll-out llamafiles for Meta's newest LLaMA 3 models-8B-Instruct and 70B-Instruct-within a day of their release. With yesterday's 0.8 release, llamafile can also run Grok, Mixtral 8x22B, and Command-R.

Creating your own llamafiles

Since the day that llamafile shipped people have wanted to create their own llamafiles. Previously, this required a number of steps, but today you can do it with a single command, e.g.:

llamafile-convert [model.gguf]

In just moments, this will produce a "model.llamafile" file that is ready for immediate use. Our thanks to community member @chan1012 for contributing this helpful improvement.

In a related development, Hugging Face recently added official support for llamafile within their model hub. This means you can now search and filter Hugging Face specifically for llamafiles created and distributed by other people in the open source community.

OpenAI-compatible API server

Since it's built on top of llama.cpp, llamafile inherits that project's server component, which provides OpenAI-compatible API endpoints. This enables developers who are building on top of OpenAI to switch to using open models instead. At Mozilla we very much want to support this kind of future: one where open-source AI is a viable alternative to centralized, closed, commercial offerings.

While open models do not yet fully rival the capabilities of closed models, they're making rapid progress. We believe that making it easier to pivot existing code over to executing against open models will increase demand and further fuel this progress.

Over the past few months, we've invested effort in extending these endpoints, both to increase functionality and improve compatibility. Today, llamafile can serve as a drop-in replacement for OpenAI in a wide variety of use cases.

We want to further extend our API server's capabilities, and we're eager to hear what developers want and need. What's holding you back from using open models? What features, capabilities, or tools do you need? Let us know!

Integrations with other open source AI projects

Finally, it's been a delight to see llamafile adopted by independent developers and integrated into leading open source AI projects (like Open Interpreter). Kudos in particular to our own Kate Silverstein who landed PRs that add llamafile support to LangChain and LlamaIndex (with AutoGPT coming soon).

If you're a maintainer or contributor to an open source AI project that you feel would benefit from llamafile integration, let us know how we can help.

Join us!

The llamafile project is just getting started, and it's also only the first step in a major new initiative on Mozilla's part to contribute to and participate in the open source AI community. We'll have more to share about that soon, but for now: I invite you to join us on the llamafile project!

The best place to connect with both the llamafile team at Mozilla and the overall llamafile community is over at our Discord server, which has a dedicated channel just for llamafile. And of course, your enhancement requests, issues, and PRs are always welcome over at our GitHub repo.

I hope you'll join us. The next few months are going to be even more interesting and unexpected than the last, both for llamafile and for open source AI itself.

The post Llamafile's progress, four months in appeared first on Mozilla Hacks - the Web developer blog.

25 Apr 2024 3:34pm GMT

24 Apr 2024

feedPlanet Mozilla

This Week In Rust: This Week in Rust 544

Hello and welcome to another issue of This Week in Rust! Rust is a programming language empowering everyone to build reliable and efficient software. This is a weekly summary of its progress and community. Want something mentioned? Tag us at @ThisWeekInRust on Twitter or @ThisWeekinRust on mastodon.social, or send us a pull request. Want to get involved? We love contributions.

This Week in Rust is openly developed on GitHub and archives can be viewed at this-week-in-rust.org. If you find any errors in this week's issue, please submit a PR.

Updates from Rust Community

Foundation
Newsletters
Project/Tooling Updates
Observations/Thoughts
Rust Walkthroughs
Research
Miscellaneous

Crate of the Week

This week's crate is scandir, a high-performance file tree scanner.

Thanks to Marty B. for the self-suggestion!

Please submit your suggestions and votes for next week!

Call for Testing

An important step for RFC implementation is for people to experiment with the implementation and give feedback, especially before stabilization. The following RFCs would benefit from user testing before moving forward:

If you are a feature implementer and would like your RFC to appear on the above list, add the new call-for-testing label to your RFC along with a comment providing testing instructions and/or guidance on which aspect(s) of the feature need testing.

Call for Participation; projects and speakers

CFP - Projects

Always wanted to contribute to open-source projects but did not know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started!

Some of these tasks may also have mentors available, visit the task page for more information.

If you are a Rust project owner and are looking for contributors, please submit tasks here.

CFP - Speakers

Are you a new or experienced speaker looking for a place to share something cool? This section highlights events that are being planned and are accepting submissions to join their event as a speaker.

If you are an event organizer hoping to expand the reach of your event, please submit a link to the submission website through a PR to TWiR.

Updates from the Rust Project

432 pull requests were merged in the last week

Rust Compiler Performance Triage

A week dominated by small mixed changes to perf with improvements slightly outweighing regressions. There were no pure regressions, and many of the mixed perf results were deemed worth it for their potential improvements to runtime performance through further optimization from LLVM.

Triage done by @rylev. Revision range: ccfcd950..a77f76e2

Summary:

(instructions:u) mean range count
Regressions ❌
(primary)		 0.4% [0.2%, 1.8%] 57
Regressions ❌
(secondary)		 0.4% [0.2%, 1.9%] 26
Improvements ✅
(primary)		 -0.8% [-3.4%, -0.2%] 50
Improvements ✅
(secondary)		 -0.6% [-1.9%, -0.1%] 32
All ❌✅ (primary) -0.2% [-3.4%, 1.8%] 107

0 Regressions, 5 Improvements, 6 Mixed; 2 of them in rollups 62 artifact comparisons made in total

Full report here

Call for Testing

An important step for RFC implementation is for people to experiment with the implementation and give feedback, especially before stabilization. The following RFCs would benefit from user testing before moving forward:

RFCs
Rust

If you are a feature implementer and would like your RFC to appear on the above list, add the new call-for-testing label to your RFC along with a comment providing testing instructions and/or guidance on which aspect(s) of the feature need testing.

Approved RFCs

Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week:

Final Comment Period

Every week, the team announces the 'final comment period' for RFCs and key PRs which are reaching a decision. Express your opinions now.

RFCs
Tracking Issues & PRs

Rust

Cargo

New and Updated RFCs

Upcoming Events

Rusty Events between 2024-04-24 - 2024-05-22 🦀

Virtual
Africa
Asia
Europe
North America
Oceania

If you are running a Rust event please add it to the calendar to get it mentioned here. Please remember to add a link to the event too. Email the Rust Community Team for access.

Jobs

Please see the latest Who's Hiring thread on r/rust

Quote of the Week

The learning curve for Rust is relatively steep compared to other languages, but once you climb it you'll never look down.

- BD103 on Mastodon

Thanks to BD103 for the self-suggestion!

Please submit quotes and vote for next week!

This Week in Rust is edited by: nellshamrell, llogiq, cdmistman, ericseppanen, extrawurst, andrewpollack, U007D, kolharsam, joelmarcey, mariannegoldin, bennyvasquez.

Email list hosting is sponsored by The Rust Foundation

Discuss on r/rust

24 Apr 2024 4:00am GMT

23 Apr 2024

feedPlanet Mozilla

Niko Matsakis: Sized, DynSized, and Unsized

Extern types have been blocked for an unreasonably long time on a fairly narrow, specialized question: Rust today divides all types into two categories - sized, whose size can be statically computed, and unsized, whose size can only be computed at runtime. But for external types what we really want is a third category, types whose size can never be known, even at runtime (in C, you can model this by defining structs with an unknown set of fields). The problem is that Rust's ?Sized notation does not naturally scale to this third case. I think it's time we fixed this. At some point I read a proposal - I no longer remember where - that seems like the obvious way forward and which I think is a win on several levels. So I thought I would take a bit of time to float the idea again, explain the tradeoffs I see with it, and explain why I think the idea is a good change.

TL;DR: write T: Unsized in place of T: ?Sized (and sometimes T: DynSized)

The basic idea is to deprecate the ?Sized notation and instead have a family of Sized supertraits. As today, the default is that every type parameter T gets a T: Sized bound unless the user explicitly chooses one of the other supertraits:

/// Types whose size is known at compilation time (statically).
/// Implemented by (e.g.) `u32`. References to `Sized` types
/// are "thin pointers" -- just a pointer.
trait Sized: DynSized { }

/// Types whose size can be computed at runtime (dynamically).
/// Implemented by (e.g.) `[u32]` or `dyn Trait`.
/// References to these types are "wide pointers",
/// with the extra metadata making it possible to compute the size
/// at runtime.
trait DynSized: Unsized { }

/// Types that may not have a knowable size at all (either statically or dynamically).
/// All types implement this, but extern types **only** implement this.
trait Unsized { }

Under this proposal, T: ?Sized notation could be converted to T: DynSized or T: Unsized. T: DynSized matches the current semantics precisely, but T: Unsized is probably what most uses actually want. This is because most users of T: ?Sized never compute the size of T but rather just refer to existing values of T by pointer.

Credit where credit is due?

For the record, this design is not my idea, but I'm not sure where I saw it. I would appreciate a link so I can properly give credit.

Why do we have a default T: Sized bound in the first place?

It's natural to wonder why we have this T: Sized default in the first place. The short version is that Rust would be very annoying to use without it. If the compiler doesn't know the size of a value at compilation time, it cannot (at least, cannot easily) generate code to do a number of common things, such as store a value of type T on the stack or have structs with fields of type T. This means that a very large fraction of generic type parameters would wind up with T: Sized.

So why the ?Sized notation?

The ?Sized notation was the result of a lot of discussion. It satisfied a number of criteria.

? signals that the bound operates in reverse

The ? is meant to signal that a bound like ?Sized actually works in reverse from a normal bound. When you have T: Clone, you are saying "type T must implement Clone". So you are narrowing the set of types that T could be: before, it could have been both types that implement Clone and those that do not. After, it can only be types that implement Clone. T: ?Sized does the reverse: before, it can only be types that implement Sized (like u32), but after, it can also be types that do not (like [u32] or dyn Debug). Hence the ?, which can be read as "maybe" - i.e., T is "maybe" Sized.

? can be extended to other default bounds

The ? notation also scales to other default traits. Although we've been reluctant to exercise this ability, we wanted to leave room to add a new default bound. This power will be needed if we ever adopt "must move" types1 or add a bound like ?Leak to signal a value that cannot be leaked.

But ? doesn't scale well to "differences in degree"

When we debated the ? notation, we thought a lot about extensibility to other orthogonal defaults (like ?Leak), but we didn't consider extending a single dimension (like Sized) to multiple levels. There is no theoretical challenge. In principle we could say…

…but I personally find that very confusing. To me, saying something "might be statically sized" does not signify that it is dynamically sized.

And ? looks "more magical" than it needs to

Despite knowing that T: ?Sized operates in reverse, I find that in practice it still feels very much like other bounds. Just like T: Debug gives the function the extra capability of generating debug info, T: ?Sized feels to me like it gives the function an extra capability: the ability to be used on unsized types. This logic is specious, these are different kinds of capabilities, but, as I said, it's how I find myself thinking about it.

Moreover, even though I know that T: ?Sized "most properly" means "a type that may or may not be Sized", I find it wind up thinking about it as "a type that is unsized", just as I think about T: Debug as a "type that is Debug". Why is that? Well, beacuse ?Sized types may be unsized, I have to treat them as if they are unsized - i.e., refer to them only by pointer. So the fact that they might also be sized isn't very relevant.

How would we use these new traits?

So if we adopted the "family of sized traits" proposal, how would we use it? Well, for starters, the size_of methods would no longer be defined as T and T: ?Sized…

fn size_of<T>() -> usize {}
fn size_of_val<T: ?Sized>(t: &T) -> usize {}

… but instead as T and T: DynSized …

fn size_of<T>() -> usize {}
fn size_of_val<T: DynSized>(t: &T) -> usize {}

That said, most uses of ?Sized today do not need to compute the size of the value, and would be better translated to Unsized…

impl<T: Unsized> Debug for &T {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) { .. }
}

Option: Defaults could also be disabled by supertraits?

As an interesting extension to today's system, we could say that every type parameter T gets an implicit Sized bound unless either…

  1. There is an explicit weaker alternative(like T: DynSized or T: Unsized);
  2. Or some other bound T: Trait has an explicit supertrait DynSized or Unsized.

This would clarify that trait aliases can be used to disable the Sized default. For example, today, one might create a Value trait is equivalent to Debug + Hash + Org, roughly like this:

trait Value: Debug + Hash + Ord {
    // Note that `Self` is the *only* type parameter that does NOT get `Sized` by default
}

impl<T: ?Sized + Debug + Hash + Ord> Value for T {}

But what if, in your particular data structure, all values are boxed and hence can be unsized. Today, you have to repeat ?Sized everywhere:

struct Tree<V: ?Sized + Value> {
    value: Box<V>,
    children: Vec<Tree<V>>,
}

impl<V: ?Sized + Value> Tree<V> { … }

With this proposal, the explicit Unsized bound could be signaled on the trait:

trait Value: Debug + Hash + Ord + Unsized {
    // Note that `Self` is the *only* type parameter that does NOT get `Sized` by default
}

impl<T: Unsized + Debug + Hash + Ord> Value for T {}

which would mean that

struct Tree<V: Value> { … }

would imply V: Unsized.

Alternatives

Different names

The name of the Unsized trait in particular is a bit odd. It means "you can treat this type as unsized", which is true of all types, but it sounds like the type is definitely unsized. I'm open to alternative names, but I haven't come up with one I like yet. Here are some alternatives and the problems with them I see:

All told, I think Unsized is the best name. It's a bit wrong, but I think you can understand it, and to me it fits the intuition I have, which is that I mark type parameters as Unsized and then I tend to just think of them as being unsized (since I have to).

Some sigil

Under this proposal, the DynSized and Unsized traits are "magic" in that explicitly declaring them as a bound has the impact of disabling a default T: Sized bound. We could signify that in their names by having their name be prefixed with some sort of sigil. I'm not really sure what that sigil would be - T: %Unsized? T: ?Unsized? It all seems unnecessary.

Drop the implicit bound altogether

The purist in me is tempted to question whether we need the default bound. Maybe in Rust 2027 we should try to drop it altogether. Then people could write

fn size_of<T: Sized>() -> usize {}
fn size_of_val<T: DynSized>(t: &T) -> usize {}

and

impl<T> Debug for &T {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) { .. }
}

Of course, it would also mean a lot of Sized bounds cropping up in surprising places. Beyond random functions, consider that every associated type today has a default Sized bound, so you would need

trait Iterator {
    type Item: Sized;
}

Overall, I doubt this idea is worth it. Not surprising: it was deemed too annoying before, and now it has the added problem of being hugely disruptive.

Conclusion

I've covered a design to move away from ?Sized bounds and towards specialized traits. There are avrious "pros and cons" to this proposal but one aspect in particular feels common to this question and many others: when do you make two "similar but different" concepts feel very different - e.g., via special syntax like T: ?Sized - and when do you make them feel very similar - e.g., via the idea of "special traits" where a bound like T: Unsized has extra meaning (disabling defaults).

There is a definite trade-off here. Distinct syntax help avoid potential confusion, but it forces people to recognize that something special is going on even when that may not be relevant or important to them. This can deter folks early on, when they are most "deter-able". I think it can also contribute to a general sense of "big-ness" that makes it feel like understanding the entire language is harder.

Over time, I've started to believe that it's generally better to make things feel similar, letting people push off the time at which they have to learn a new concept. In this case, this lessens my fears around the idea that Unsized and DynSized traits would be confusing because they behave differently than other traits. In this particular case, I also feel that ?Sized doesn't "scale well" to default bounds where you want to pick from one of many options, so it's kind of the worst of both worlds - distinct syntax that shouts at you but which also fails to add clarity.

Ultimately, though, I'm not wedded to this idea, but I am interested in kicking off a discussion of how we can unblock extern types. I think by now we've no doubt covered the space pretty well and we should pick a direction and go for it (or else just give up on extern types).

  1. I still think "must move" types are a good idea - but that's a topic for another post. ↩︎

23 Apr 2024 8:51pm GMT