December 7 - It's time for the Friday Highlight! Head over here to check it out!
November 20 - It's Patch Day for The Sims 4 on PC/MAC. Patch notes here.

Python Upgrade Coming Soon

«1
SimGuruModSquadSimGuruModSquad Posts: 596 SimGuru
Hi everyone,

I want to give you a heads up that in an upcoming release, we will be upgrading the version of Python used by the game. We currently use Python 3.3.5 and will be moving to Python 3.7.0. Once this change is released, all compiled script mods will completely stop working. Note that in this new version of Python, the .pyo extension is replaced by .pyc, see https://www.python.org/dev/peps/pep-0488/

We know this will have impact on your script mod authoring pipelines so wanted to provide notice in advance so you can be ready for the change when it arrives. To assist, I’ve attached the Python scripts from the currently released version of the game (v1.46.18) recompiled under Python 3.7.0.

If all goes as planned, this change should arrive in November.

Note that there will not be any fundamental changes to tuning data, other than the typical types of changes we make to tuning for patches.

Thanks!
SGMS
Post edited by SimGuruModSquad on

Comments

  • TURBODRIVERTURBODRIVER Posts: 30 Member
    edited October 3
    Oh boy :# Thanks for the heads up!
    Sorry, but not sorry tumblr_paqcr5LaDB1vwgrfio1_r1_75sq.png
  • ScumbumboScumbumbo Posts: 139 Member
    Short term this is going to be such a headache, but I think in the long run this is really good news. Kind of ironic though that I was just telling someone recently that EA would likely never update the Python version due to the number of mods it would break!

    As Turbo says, THANKS FOR THE WARNING and for including those recompiled scripts so we can try and figure out how we're going to reverse engineer those in advance. :)

    Any chance we get a more concrete idea of the time frame as you know more about when this bomb will drop?
  • LittleMsSamLittleMsSam Posts: 52 Member
    edited October 3
    Oh my^^ Ok i am no script modder at all and i use a *.bat File to compile py to pyo. I want to be ready when the time comes to be able to update my mods asap :P

    Anyone know if i can still use the bat if i change it to this after upgrading phyton to 3.7.0?:
    @echo off
    if %1x==x goto usage
    %~d1
    cd "%~p1"
    echo on
    python -O -c "import py_compile;py_compile.compile('%~n1%~x1','%~n1.pyc')"
    @echo off
    goto end
    :usage
    echo drag+drop the .py file onto this to compile to a .pyc file.
    :end
    pause
  • TURBODRIVERTURBODRIVER Posts: 30 Member
    It's very unfortunate to see that this could be the death of many great new script mods. The lack of a reliable decompiler for Python 3.7 means that a lot of essential sources are missing and without them there is no way to figure out how many elements of the game code work.

    Is it possible that there is some documentation that we can get that explains all of the files and functions? Is there anything that can help us now @SimGuruModSquad?
    Sorry, but not sorry tumblr_paqcr5LaDB1vwgrfio1_r1_75sq.png
  • ScumbumboScumbumbo Posts: 139 Member
    @LittleMsSam Compiling existing scripts won't be an issue, there will be some changes required to some batch files but that's minor.

    @SimGuruModSquad The problem will be getting the EA scripts we refer to for new mods. So far the best decompiler we've been able to work with for Python 3.7 is uncompyle6 and that just chokes and dies on any seriously important script files. Many critical files that would be a required reference for a script modder are simply no longer accessible (unless someone finds a better uncompiler, but they are mostly unmaintained since Python 3.5 bytecode changes).

    game_object.py, object_tests.py, sim_info.py and many other critical files we'd need to reference are simply cannot be decompiled at this point. Unless something changes, this may well spend the end of script modding for me personally and likely others - I for one have no desire to continue to support mods when I can't look at script changes that break those mods. And certainly any new scripting would be unthinkable for me.

    Unless somehow EA can reverse their viewpoint on releasing the core scripts to modders for reference, or we can find a reliable method to reverse engineer those files - something which EA has been aware of us doing and quietly approved of since the game came out.
  • TwistedMexicanTwistedMexican Posts: 648 Member
    I agree with @Scumbumbo if you're going forward with this change we need to have access to the source.

    When Sims 4 dropped it toted itself as being modder friendly, and in many ways it is. But other ways it's frustratingly restrictive for the most arbitrary reasons.

    If we're able to get the tuning source, why not the python source as well? Without it, this is practically a move to end script mods, whether that's the actual reason or not.
    [Sims 4 Cheat Resource] || [My Mods - Base Game] || [My Mods - Expansions]
    [Lifetime Skills || Set Age || Full House ||Persistent FullEditMode || Persistent TestingCheats || DayWalker Vampires]

    I'm now on Twitter. I'd love it if you joined me there :)
  • PIsForPerryPIsForPerry Posts: 1 New Member
    Not a modder, so maybe this doesn’t “count” but I’ve been with SIMS since day one. If those of us who rely on Mods to make our games fun lose out because they can’t fix the scripts easily, you’ll see a drastic drop in gameplay. I love the work ya’ll put into this game, and the work the modders put in too. Please make it easy for them to fix this issue when you release the update!
  • Evie0602Evie0602 Posts: 486 Member
    edited October 4
    I'm not a modder either, but this is really worrying me. I use so many mods and I can't play my game without them.
    Post edited by Evie0602 on
  • ScumbumboScumbumbo Posts: 139 Member
    Do not panic yet, folks! EA notified us of this in advance because there were bound to be unforseen issues. We have to give them time to digest what those issues are and determine what their best response to the issue is. They're being proactive for a reason!
  • stilljustme2stilljustme2 Posts: 23,142 Member
    I've been cutting back on script mod usage -- pretty much rely on MCCC and UI Cheats though for both gameplay and townie creation/fine tuning. It's been great going in and easily adding extra traits like the ones from Parenthood to fill out personalities on the premades. I hope that the change in Python version doesn't kill those -- as @PIsForPerry said, there'll be a lot of players who'll give up on the game if they lose script mods especially MCCC. Posted the link to this on the MCCC Discord server to give @Deaderpool a heads up.
    Check out my Gallery! Origin ID: justme22
    Fun must be always -- Tomas Hertl (San Jose Sharks hockey player)
  • QueenMercyQueenMercy Posts: 671 Member
    He already knows. He sent out an email to his patreon subscribers like an hour ago.
  • DeaderpoolDeaderpool Posts: 186 Member
    Yep. Aware of it and, like has been said here previously, I'm just hoping that this means that going forward EA will give us the decompiled scripts like they did on this post when a new release comes out. We've always had to decompile the core-game scripts to do script modding. If going to the new version breaks our decompiler (which it does), then it would be nice to know that they're going to take care of us going forward by providing that.

    Otherwise, compiling to a new version and all, we can make work. Those of us still around, anyway. ;)
  • stilljustme2stilljustme2 Posts: 23,142 Member
    QueenMercy wrote: »
    He already knows. He sent out an email to his patreon subscribers like an hour ago.

    Hadn't checked email since this morning -- I'm a Patreon though I had to step back to first tier due to some financial issues (hope to be back up to second tier in the next couple of months).
    Check out my Gallery! Origin ID: justme22
    Fun must be always -- Tomas Hertl (San Jose Sharks hockey player)
  • TwistedMexicanTwistedMexican Posts: 648 Member
    Deaderpool wrote: »
    Yep. Aware of it and, like has been said here previously, I'm just hoping that this means that going forward EA will give us the decompiled scripts like they did on this post when a new release comes out. We've always had to decompile the core-game scripts to do script modding. If going to the new version breaks our decompiler (which it does), then it would be nice to know that they're going to take care of us going forward by providing that.

    Otherwise, compiling to a new version and all, we can make work. Those of us still around, anyway. ;)

    Small correction, the file sgms posted here is still compiled code. It remains to be seen if they'll provide uncompiled code but here's hoping. 🤞
    [Sims 4 Cheat Resource] || [My Mods - Base Game] || [My Mods - Expansions]
    [Lifetime Skills || Set Age || Full House ||Persistent FullEditMode || Persistent TestingCheats || DayWalker Vampires]

    I'm now on Twitter. I'd love it if you joined me there :)
  • luthienrisingluthienrising Posts: 30,508 Member
    Modders will update things @MzNatNat , same as always when patches break them. And they have lots of notice :)
    FAQ: culling relationship decay
    #freethebabies #TeamToasters #TeamPlayableGnomes #TeamFarms #FarmTeam #BouncyCastle!!
  • paradiseplanetparadiseplanet Posts: 3,721 Member
    edited October 4
    It's MODageddon!!!

    But no really, I have hopes that the super awesome modding community can work their way even out of this one...

    I hope.
    Post edited by paradiseplanet on
    Origin ID: paradiseplanet27
    Simblr

    tumblr_ojq4r339Ni1usy5rpo1_100.png
  • DeaderpoolDeaderpool Posts: 186 Member
    Deaderpool wrote: »
    Yep. Aware of it and, like has been said here previously, I'm just hoping that this means that going forward EA will give us the decompiled scripts like they did on this post when a new release comes out. We've always had to decompile the core-game scripts to do script modding. If going to the new version breaks our decompiler (which it does), then it would be nice to know that they're going to take care of us going forward by providing that.

    Otherwise, compiling to a new version and all, we can make work. Those of us still around, anyway. ;)

    Small correction, the file sgms posted here is still compiled code. It remains to be seen if they'll provide uncompiled code but here's hoping. 🤞

    Nice to know. I haven't unzipped it or done anything with it yet. Planned to do that tomorrow. Thanks!
  • egwarhammeregwarhammer Posts: 5,748 Member
    Modders are preparing, and waiting to see what happens. I suggest players follow their lead, here.

    IMO, "potential fallout" (from the players perspective) would best be discussed in other threads besides this one.

    Let's leave this one for the Coding discussions.

    You could join us on Deaderpool's Discord, too. Hot Topic in #creators_workshop there, for sure.
  • FogityFogity Posts: 122 Member
    @SimGuruModSquad
    If you still aren't willing to give us the uncompiled script sources, could you reconsider giving us the documentation?
    I am assuming that you generate documentation for internal use, and I can't see how it would be worse than giving us the TDESC's.
    Although I haven't delved much into script modding myself, my guess is that having access to the documentation is far more useful to us than having access to decompiled code.

    @Scumbumbo
    Remind me if I'm wrong, but weren't we unable to decompile a lot of files in the beginning, but then someone went in and fixed a decompiler?
    We probably could look into why uncompyle6 chokes on the files and maybe fix the problem there.
    As our community uses these tools so much it would be nice if we gave back some, if we can that is.

    To everyone else:
    I'd suggest that anyone that Isn't going to be working on decompiling the scripts keep from posting here, not because your opinions aren't valid, but because if you derail the conversation it's less likely that we will come to a good solution. Consider making a new thread instead.
    Played since The Sims 1 and now makes mods for The Sims 4:
    Gender Less - Gender neutral pronouns and titles
    Hidden Highlight - Removes the white outline
    Unlocked CAS Parts - Unlocks hidden CAS items
  • TURBODRIVERTURBODRIVER Posts: 30 Member
    Fogity wrote: »
    @SimGuruModSquad
    If you still aren't willing to give us the uncompiled script sources, could you reconsider giving us the documentation?
    I am assuming that you generate documentation for internal use, and I can't see how it would be worse than giving us the TDESC's.
    Although I haven't delved much into script modding myself, my guess is that having access to the documentation is far more useful to us than having access to decompiled code.

    If not decompiled code, the best they can do is give us an outline of the code (by wiping the actual code) with the developer notes. But something tells me that there might not be any useful (for us) notes in there. I might be wrong about it but the programming staff hasn't changed much for the last 4 years which means it's a very minimal documentation. Looks like a lot of sections of the sources is dealt with by individual programmers since they include their names there, so they don't need that much documentation in the first place.
    And we can generate an outline like that ourselves, but it won't be anywhere near as helpful as the actual source.
    Fogity wrote: »
    Remind me if I'm wrong, but weren't we unable to decompile a lot of files in the beginning, but then someone went in and fixed a decompiler?
    We probably could look into why uncompyle6 chokes on the files and maybe fix the problem there.
    As our community uses these tools so much it would be nice if we gave back some, if we can that is.

    The decompiler for Python 3.3 was actually designed for 3.2 but then edited for someone somewhere to work with 3.3. I don't think it was the Sims community that got it to work, it was the case of using the decomplier for the right version since they are named the same. And if I'm wrong, just by looking at uncompyle6, we are dealing with something a lot more complex and editing will be challenging.
    Sorry, but not sorry tumblr_paqcr5LaDB1vwgrfio1_r1_75sq.png
  • coolspearcoolspear Posts: 23 Member
    edited October 4
    Oh, man. And I was just getting into scripting. Swatting up and slowly botching a few attempts. Well scrap that now for a pointless endeavour. Locked in tuning purgatory for a while longer, I guess.

    Well, as Scumbumbo pointed out, early days yet, EA putting out the feelers, and thus may well tune their final approach to fully consider the script modders who have put so much hard work in over these last four years.

    I imagine healthy game sales figures do depend on it. Don't bite the hands that feed you, nor the ones holding up your chair. So...
  • ScumbumboScumbumbo Posts: 139 Member
    edited October 4
    @fogity That "fix" you are thinking of back then for unpyc3 was iirc a pretty simple one, just ignored an error by trapping the exception and moving on as best it could. Produced some gibberish, but hey that's life.

    The complex fixes were, as Turbo mentions above, done by someone outside TS4 modding community.

    Andrew and I have both tried some similar hacks in the same vein without any really notable results.

    I'm going to run everything through decompilation tomorrow and summarize exactly what results we end up with and provide those to SGMS. Andrew and I were just chatting with them on Discord, I won't post the gist of the convo here as I don't want to put words in EA's mouth obviously. All I can say here is that releasing sources is unlikely (as we expected), but they seem committed to helping us work out issues with decompilation.

    At a guess, I'd say roughly 75% of files seem to uncompile properly using uncompyle6, but those that fail are quite important ones as a general rule. Some that uncompile have some invalid syntax in them, particularly those with complex nested expressions. My tests I run will involve recompiling the uncompiled sources to check their validity.

    Feel free to PM me on DP's Discord channel if you want some more unofficial info, I imagine SGMS will be posting here as well once anything is official.

    @SimGuruModSquad - thanks a lot for chatting with us, it is good to know definitively that you will assist in-so-far as you are able to.
  • FogityFogity Posts: 122 Member
    @Scumbumbo
    Although I really shouldn't be working on the sims, I have fallen down the rabbit hole again. I might be interested in looking into trying to fix the issues in uncompyle6. Although python is not my language of choice I do have knowledge about compilers and such.
    Played since The Sims 1 and now makes mods for The Sims 4:
    Gender Less - Gender neutral pronouns and titles
    Hidden Highlight - Removes the white outline
    Unlocked CAS Parts - Unlocks hidden CAS items
  • ScumbumboScumbumbo Posts: 139 Member
    @simgurumodsquad I have those results you wanted for the uncompyle6 tests. I just ran it on the files in simulation and core as we don't really care about base or generated.

    1,261 (73%) files uncompiled and tested 100% accurate.
    296 (17%) files uncompoiled but could not be recompiled due to syntax errors.
    169 (10%) files threw exceptions during the uncompile run, of those 10 were due to pressing Ctrl C as those files would get stuck in an infinite loop. The remaining 159, at least all the ones I saw, threw the assert failure due to not being able to properly parse the kwargs ending up with a negative positional arg count.

    I can send you a copy of the results, which would include the failed uncompiled files and the log that was generated with all exceptions and messages about failed verifications, just let me know how you want me to get that to you.

    Meanwhile, Fogity and Turbo have been trying some stuff with pycdc and I'll likely try again to get a working copy of that compiled so I can look into that.
  • PinkusfamilyPinkusfamily Posts: 210 Member
    Right now I am Modless and CC-less. But in the past I had a lot of CC and even dabbled in mods. So I have faith in the modders and cc makers- they are an intelligent & creative group, and I think they will adapt..On another subject This change I believe has to do with the whole new terrain changing tool.I don't know much technical stuff But I hope this leads to us being able to make our own Worlds/neighborhoods- like in Sims 3 with the Create a world tool...I have faith in our Modders they will adapt !!

    Thanks,
    Diane Pinkus
«1
Sign In or Register to comment.
Return to top