The problem right now is basically that each square is a new surface we are filling, and we are redrawing every surface every time
No, it's not. Each square is a rect filled directly into the main rendering surface. From what I'm reading in the pygame docs, the current method is at least as fast as pygame.draw.*.
No, it's not. Each square is a rect filled directly into the main rendering surface. From what I'm reading in the pygame docs, the current method is at least as fast as pygame.draw.*.
The problem right now is basically that each square is a new surface we are filling, and we are redrawing every surface every time
No, it's not. Each square is a rect filled directly into the main rendering surface. From what I'm reading in the pygame docs, the current method is at least as fast as pygame.draw.*.
My thinking is that rather than constantly redraw each square, draw the terrain to a single or set of surfaces(one per useful elevation level for example), that we update as necessary. Then we wouldn't have to redraw many surfaces and only the single surface with resizing, shifting, etc.
I have made a huge improvement to the terrain rendering, using some of the approaches described above. Here's how it works - there is, as discussed, a single (invisible) surface that holds a full image of the generated terrain, with each terrain square taking up a single pixel. This surface is generated only once, before the game starts.
ViewPort now has a render_terrain method, which works as follows: 1)the visible subset of the terrain is selected using pygame.Surface.subsurface 2)this visible subset is scaled to the visible window size using pygame.transform.scale I was considering storing the terrain surface and only updating it when the view changes, but I found that this approach is so fast that the additional overhead of storing and copying the terrain actually made it slower.
Since it's a non-sampling scale function, it won't do any blurring between grid squares, but if the resolution isn't quite divisible by the block size I guess some blocks would be different in size by a pixel or two. If this is a problem, feel free to change the zoom mechanics to ensure divisibility. Oh, and I set the repeat interval to 1ms and forgot to change it back.
Changing z-levels no longer functions as it used to. Perhaps we should generate a surface for each level, and then going up and down through the z-axis would change which surface the viewport would take a subsurface from?
If we maintain the Dwarf Fortress style z-level changing, then we do need to generate a separate surface view for each z-level. We basically have to generate a 2D surface image for each Z and also a 3D graph of the terrain to match it.
I think a better idea would be instead to simply have the entire thing be in 3D. Changing the Z-level will be like using a wall-hack almost. If you set it all the way at the top, then you will only see the outside of the mountains, and all the things on the surface. If you set it to be deep, then everything above that Z-level won't render, and you can look inside. Think of it like an ant farm.
If I knew people were actually going to work on this, we should have done a Battle Balls clone for XNA instead. We could have made moneys with that one.
Even so, I think I'm going to make a patch to start moving things towards an XNA-like architecture. It's actually also a pygame-like architecture.
Basically, you make (almost) every class a sub-class of GameComponent, or equivalent. Every GameComponent implements methods such as __init__, update, render, etc. Then you register or unregister instances of GameComponent subclasses. If a component instance is registered, it will receive input, update itself every time through the game loop, render, etc.
I'm also going to do some python path finagling to organize our files in a better way.
I would have no problem with XNA, though I figured we were using Python to make things easier for the newb forumites.
I've set the key repeating to a delay of 0.5s and a repeat interval of 33ms, which seems pretty decent (though I'm still wondering about what would be optimal). To allow the movers to make use of the maximum framerate, the "t" key now toggles between automatic (every frame) and manual (space-activated) moving of the movers.
Since it's a non-sampling scale function, it won't do any blurring between grid squares, but if the resolution isn't quite divisible by the block size I guess some blocks would be different in size by a pixel or two. If this is a problem, feel free to change the zoom mechanics to ensure divisibility.
I've found the obvious problem with this approach - while the terrain looks fine even when the scaling isn't exact, it throws off the co-ordinates. You can see this problem if you create movers and choose one of the intermediate zoom levels.
Another problem I've found, which should have been obvious earlier, is that the game generates the same terrain every time it's run. I'm guessing that this is because we're not seeding the random number generator. In my opinion, the seed should probably be an input to the generators to ensure repeatability. On a similar note, now that Andrew has implemented saving and loading the grid, we should offer the option of loading on startup before a new grid is generated.
I've added these to the issues. Does anyone want to work on any of these in particular? Fixing something could be a good way to learn if you're one of the people currently lurking on this project
The next chance I get I'm going to make something we sorely need, a main menu. That way the game will start, and you can generate a terrain only if you want to, and load any of multiple saved terrains. This is much needed for testing purposes as well. It's very annoying to have to regen a terrain every time you load the game just to test something else.
For example, lackofcheese might do this to be able to fetch amelim’s changes that are not yet available at upstream. $ git remote add amelim git://github.com/amelim/Project-DORF.git $ git fetch amelim $ git merge amelim/master
Indeed, lackofcheese has done exactly that before, down to the local name itself.
Thanks for the post, though. I haven't had the occasion to use rebase yet, but I'm going to have to look into it now. Also, now I understand why your main branch kept having commits that were duplicates of those in other people's code - that's the effect of rebasing.
I'd like to hear your thoughts on rebasing vs merging; there's an interesting article on the matter here. It seems that the difference is that rebase will create duplicates of the commits of one branch in the history of the other, while merge creates a single commit in one branch whose history references both of the branches it came from.
I'd like to hear your thoughts on rebasing vs merging; there's an interesting article on the matterhere.
I really don't understand these people who think it's a versus situation. Discussing merging versus rebasing, as if one were somehow better, is like discussing hammers vs. screwdrivers.
Basically, the idea is that you always want your changes to be on top of changes that have come before. For example. There is a master branch with three commits.
master - A - B - C
You branch off at point C and start working and you make commits X Y Z
mybranch - A B C X Y Z
Meanwhile, someone adds D and E to the master branch.
master - A B C D E mybranch - A B C X Y Z
If you just go ahead and merge mybranch into master, then who knows what kind of conflict there will be. Your XYZ doesn't take into account D and E. So the first thing to do is to rebase mybranch onto master. This will put XYZ in a separate holding bin put D and E on mybranch, then put XYZ back on top. You originally branched mybranch at point C, but after a rebase it will be as if you branched at point E.
* after rebase master - A B C D E mybranch - A B C D E X Y Z
Aha! Now it is all nice. If the rebase caused trouble, we can fix that trouble in our separate branch, keeping the master branch smooth and deadly. When we know it's good, for real, we can merge mybranch into master which will make them identical.
master - A B C D E X Y Z mybranch - A B C D E X Y Z
And then we can push the master out to the world, or request others to pull from it. They will in turn have X Y Z showing up in their master branches, and they will rebase their local branches on top of XYZ and so on.
There's also one more handy trick to know. git pull --rebase
Let's say you do a commit directly on master branch. pull is fetch merge. If you pull onto a branch that has an un-pushed commit, you might have some trouble. Doing --rebase on the pull will put your unpushed commits on top of the commits you are pulling.
The thing with rebase, is that the very same command which does this useful thing, also has other powers. For example, if you do the interactive rebase, you can basically rewrite all of history. You can do things like completely reordering sets of commits, or squashing multiple commits together into individual ones. For organizational sake, this is sometimes something you want to do, but it is rare. It's the kind of thing that if you don't think you need to do it before you know it's possible, then don't do it. If you learn that stuff, you might gain magical powers, but you just might hurt yourself instead.
Also, rebase is not the cause of any weirdness you were seeing. That was entirely because I didn't realize GitHub's fork queue web interface works via cherry picking. I'm just not going to use that anymore.
They're still both different was of combining two sets of changes, though they differ in the way they represent the history, so it makes sense to ask in what situations you want one or the other. I think the example you posted is actually a situation where you wouldn't want to use rebase the way you did.
Let me annotate your post, assuming that beforehand we have master - A B C lackofcheese/master - A B C D E F amelim/master - A B C X mycode - A B C (Z not commited yet) $ echo "These remote add commands only need to be done once ever per clone" $ git remote add ameleim git://github.com/amelim/Project-DORF.git $ git remote add lackofcheese git://github.com/lackofcheese/Project-DORF.git $ git checkout -b mycode $ git add somefiles $ git commit mycode = A B C Z $ git checkout -b cheese $ git fetch lackofcheese $ git merge lackofcheese/master cheese = A B C D E F $ git checkout -b melim $ git fetch amelim $ git merge amelim/master melim = A B C X $ git rebase cheese melim = A B C D E F X' $ echo "The melim local branch now has amelims changes merged with lackofcheese's" $ git checkout mycode $ git rebase melim mycode = A B C D E F X' Z' $ git checkout master $ git merge mycode master = A B C D E F X' Z' $ git push $ echo "mycode on top of amelim's on top of lackofcheese's pushed out to the world" $ echo "let's delete the branches we are done with $ git branch -D mycode $ git branch -D cheese $ git branch -D melimThis is the problem that was mentioned in the article I linked - X' and Z' are not the same as the original X and Z. What if amelim has the following branch: bugfix = A B C X Y and pulls the latest version of your code like this:$git fetch upstream upstream/master = A B C D E F X' Z' $git checkout bugfix $git rebase upstream/masterWouldn't the result be A B C D E F X' Z' X Y, which is an obvious conflict?
As such, I would think that you shouldn't rebase any commits that were published by others. Rebasing makes good sense if you're working on a separate feature that you're not pushing yet, and you want to update your work in progress, however.
Also, rebase is not the cause of any weirdness you were seeing. That was entirely because I didn't realize GitHub's fork queue web interface works via cherry picking. I'm just not going to use that anymore.
Are you sure? For example, there are currently three versions of the following commit: 1 2 3 What would cause duplicate commits, if not rebasing? I should think that we want to avoid duplicate commits of this nature.
What would cause duplicate commits, if not rebasing? I should think that we want to avoid duplicate commits of this nature.
Yeah, the reason those duplicate commits are there because I didn't realize that GitHub's fork queue was stupid. Basically what happens is two different people other than me have forks. They have been working together, and have shared patches with each other. So they both have a commit that is not in my repo. If you go in the fork queue it will show the same commit available from both forks. If you use the fork queue interface, it will cherry pick those commits, and it can possibly cherry pick the same commit twice, once each from two different forks.
This is the problem that was mentioned in the article I linked - X' and Z' are not the same as the original X and Z. What if amelim has the following branch: bugfix = A B C X Y and pulls the latest version of your code like this:$git fetch upstream upstream/master = A B C D E F X' Z' $git checkout bugfix $git rebase upstream/masterWouldn't the result be A B C D E F X' Z' X Y, which is an obvious conflict?
Whether or not there is a conflict really depends upon exactly which code was edited. If those XYZX'Z' commits are all on completely different files, or different parts of files, it will most likely apply perfectly cleanly. If there's any conflict, that same conflict would have also occurred if you had done a merge. If two people work on conflicting code, you have to fix it either way. Using merge instead of rebase doesn't make that any different.
The reason I like to use rebase in these cases is because it is a first-come first-served kind of thing. If someone else gets their commits in first, and you are committing after them, then it's up to you to make sure your stuff works on top of their stuff. They were in first, their code is out there, you have to make yours work on top of theirs. Using rebase makes sure your commits come after all the other ones that came before you. Then everyone else who comes after you has to make sure their stuff works on top of yours.
Also, some of these crazy conflicts only occur when you start fetching stuff from multiple collaborators. I suggest that unless you really know what you are doing, that you only ever fetch from one upstream source, which would be me. If you always make sure your master branch is tracked to my master, and you rebase all of your stuff on top of it before making a pull request, then no weirdness like that can ever occur. The weirdness starts to happen when multiple people are sharing in different directions. For example.
my master - A B C you master - A B C D him master - A B C E You and him share with each other and each do a little more work. my master - A B C you master - A B C D E F him master - A B C E D G Then you both send me pull requests. Ah crap! If you don't know what you are doing, I highly suggest you only fetch from one place and rebase on top of that. my master - A B C you master - A B C D him master - A B C E I pull in changes as requested, and everyone else starts doing new work rebased on top of my master. my master - A B C D E you master - A B C D E F him master - A B C D E G
Also, I'm going to have to learn some of the more advanced tools, because I really should squash some of those commits I did previously. EDIT: Ooh, I did it, but git rebase -i is scary.
Also, I'd like to point out another possible work pattern. Two people want to work on a particular feature together. What do they do? They should both create a separate branch, not just locally on their machines, but on GitHub. They can then work together on the feature in that separate branch. They can keep pulling each others changes just on that separate branch. Then when it's all good, one of them can merge it back into master, make a pull request, and it will disseminate.
I think a better idea would be instead to simply have the entire thing be in 3D. Changing the Z-level will be like using a wall-hack almost. If you set it all the way at the top, then you will only see the outside of the mountains, and all the things on the surface. If you set it to be deep, then everything above that Z-level won't render, and you can look inside. Think of it like an ant farm.
Between this and the isometric view, we should really consider a proper 3D engine (I feel like hacking pseudo-3D into pygame is going to get ugly, fast). I'm not sure what's good in the python space in that regard, but I'll look into it a bit.
I think a better idea would be instead to simply have the entire thing be in 3D. Changing the Z-level will be like using a wall-hack almost. If you set it all the way at the top, then you will only see the outside of the mountains, and all the things on the surface. If you set it to be deep, then everything above that Z-level won't render, and you can look inside. Think of it like an ant farm.
Between this and the isometric view, we should really consider a proper 3D engine (I feel like hacking pseudo-3D into pygame is going to get ugly, fast). I'm not sure what's good in the python space in that regard, but I'll look into it a bit.
It raises a good question, one which I don't think we've had any real direction on in the project so far. Which of the following do we want to aim for and how exactly should they function?
If my opinion matters any, I think the 2D with sprites would be best. Although the isometric looks gorgeous, I think that got ruled out as a coding issue earlier in the thread. Besides, if we went with the second option, I think more people would be able to contribute on the art side of things who can't on the coding side of things.
Well, if we just want this to be happy coding fun time, we can do whatever we want. If we want some sort of "success" I think the best way to go is to make something that looks like a well polished 2D iPhone game. Even if we do the latter, we can still use some 3D bits to make certain components of the game easier to program, faster, GPU-accelerated, etc. Think like Civilization. It's a 2D game, but they use 3D, so it's easy to zoom, pan, rotated, etc. Otherwise, there's nothing that's really 3D about it.
Also, a lot of the 3D engines out there are really geared toward a more traditional idea of a 3D game. Their great for an FPS or a 3rd person shooter, but for a sim? I don't know.
We also have to consider the user interface. It's what DF is the worst at, so it's easy to beat it, but we should really hit it out of the park to deal a crushing blow. If I were playing DF I would think an isometric or actual 3D view, like the last two example screen shots, would make for a worse UI as you would have to rotate the camera often. I feel it should be more like Sim City, where you can rotate to four different angles, but you never really need to.
Also, I think we should do multiplayer at some point. If we made it work, we would kill. We don't have to do it right away, but we should code with future multiplayer in mind.
Also, I think we should do multiplayer at some point. If we made it work, we would kill. We don't have to do it right away, but we should code with future multiplayer in mind.
Well, exactly what kind of multiplayer are we looking for? Personally, I think collaborative fortress playing should be the priority. I think multiple people working on a single fortress could work really well.
Comments
ViewPort now has a render_terrain method, which works as follows:
1)the visible subset of the terrain is selected using pygame.Surface.subsurface
2)this visible subset is scaled to the visible window size using pygame.transform.scale
I was considering storing the terrain surface and only updating it when the view changes, but I found that this approach is so fast that the additional overhead of storing and copying the terrain actually made it slower.
Since it's a non-sampling scale function, it won't do any blurring between grid squares, but if the resolution isn't quite divisible by the block size I guess some blocks would be different in size by a pixel or two. If this is a problem, feel free to change the zoom mechanics to ensure divisibility. Oh, and I set the repeat interval to 1ms and forgot to change it back.
So, do I win?
I think a better idea would be instead to simply have the entire thing be in 3D. Changing the Z-level will be like using a wall-hack almost. If you set it all the way at the top, then you will only see the outside of the mountains, and all the things on the surface. If you set it to be deep, then everything above that Z-level won't render, and you can look inside. Think of it like an ant farm.
Even so, I think I'm going to make a patch to start moving things towards an XNA-like architecture. It's actually also a pygame-like architecture.
Basically, you make (almost) every class a sub-class of GameComponent, or equivalent. Every GameComponent implements methods such as __init__, update, render, etc. Then you register or unregister instances of GameComponent subclasses. If a component instance is registered, it will receive input, update itself every time through the game loop, render, etc.
I'm also going to do some python path finagling to organize our files in a better way.
I've set the key repeating to a delay of 0.5s and a repeat interval of 33ms, which seems pretty decent (though I'm still wondering about what would be optimal). To allow the movers to make use of the maximum framerate, the "t" key now toggles between automatic (every frame) and manual (space-activated) moving of the movers. I've found the obvious problem with this approach - while the terrain looks fine even when the scaling isn't exact, it throws off the co-ordinates. You can see this problem if you create movers and choose one of the intermediate zoom levels.
Another problem I've found, which should have been obvious earlier, is that the game generates the same terrain every time it's run. I'm guessing that this is because we're not seeding the random number generator. In my opinion, the seed should probably be an input to the generators to ensure repeatability. On a similar note, now that Andrew has implemented saving and loading the grid, we should offer the option of loading on startup before a new grid is generated.
I've added these to the issues. Does anyone want to work on any of these in particular? Fixing something could be a good way to learn if you're one of the people currently lurking on this project
GitHub Suggested Workflow
Thanks for the post, though. I haven't had the occasion to use rebase yet, but I'm going to have to look into it now. Also, now I understand why your main branch kept having commits that were duplicates of those in other people's code - that's the effect of rebasing.
I'd like to hear your thoughts on rebasing vs merging; there's an interesting article on the matter here. It seems that the difference is that rebase will create duplicates of the commits of one branch in the history of the other, while merge creates a single commit in one branch whose history references both of the branches it came from.
Basically, the idea is that you always want your changes to be on top of changes that have come before. For example. There is a master branch with three commits.
master - A - B - C
You branch off at point C and start working and you make commits X Y Z
mybranch - A B C X Y Z
Meanwhile, someone adds D and E to the master branch.
master - A B C D E
mybranch - A B C X Y Z
If you just go ahead and merge mybranch into master, then who knows what kind of conflict there will be. Your XYZ doesn't take into account D and E. So the first thing to do is to rebase mybranch onto master. This will put XYZ in a separate holding bin put D and E on mybranch, then put XYZ back on top. You originally branched mybranch at point C, but after a rebase it will be as if you branched at point E.
* after rebase
master - A B C D E
mybranch - A B C D E X Y Z
Aha! Now it is all nice. If the rebase caused trouble, we can fix that trouble in our separate branch, keeping the master branch smooth and deadly. When we know it's good, for real, we can merge mybranch into master which will make them identical.
master - A B C D E X Y Z
mybranch - A B C D E X Y Z
And then we can push the master out to the world, or request others to pull from it. They will in turn have X Y Z showing up in their master branches, and they will rebase their local branches on top of XYZ and so on.
There's also one more handy trick to know. git pull --rebase
Let's say you do a commit directly on master branch. pull is fetch merge. If you pull onto a branch that has an un-pushed commit, you might have some trouble. Doing --rebase on the pull will put your unpushed commits on top of the commits you are pulling.
The thing with rebase, is that the very same command which does this useful thing, also has other powers. For example, if you do the interactive rebase, you can basically rewrite all of history. You can do things like completely reordering sets of commits, or squashing multiple commits together into individual ones. For organizational sake, this is sometimes something you want to do, but it is rare. It's the kind of thing that if you don't think you need to do it before you know it's possible, then don't do it. If you learn that stuff, you might gain magical powers, but you just might hurt yourself instead.
Let me annotate your post, assuming that beforehand we have
master - A B C
lackofcheese/master - A B C D E F
amelim/master - A B C X
mycode - A B C (Z not commited yet)
$ echo "These remote add commands only need to be done once ever per clone"This is the problem that was mentioned in the article I linked - X' and Z' are not the same as the original X and Z.$ git remote add ameleim git://github.com/amelim/Project-DORF.git
$ git remote add lackofcheese git://github.com/lackofcheese/Project-DORF.git
$ git checkout -b mycode
$ git add somefiles
$ git commit
mycode = A B C Z
$ git checkout -b cheese
$ git fetch lackofcheese
$ git merge lackofcheese/master
cheese = A B C D E F
$ git checkout -b melim
$ git fetch amelim
$ git merge amelim/master
melim = A B C X
$ git rebase cheese
melim = A B C D E F X'
$ echo "The melim local branch now has amelims changes merged with lackofcheese's"
$ git checkout mycode
$ git rebase melim
mycode = A B C D E F X' Z'
$ git checkout master
$ git merge mycode
master = A B C D E F X' Z'
$ git push
$ echo "mycode on top of amelim's on top of lackofcheese's pushed out to the world"
$ echo "let's delete the branches we are done with
$ git branch -D mycode
$ git branch -D cheese
$ git branch -D melim
What if amelim has the following branch:
bugfix = A B C X Y
and pulls the latest version of your code like this:
$git fetch upstreamWouldn't the result be A B C D E F X' Z' X Y, which is an obvious conflict?upstream/master = A B C D E F X' Z'
$git checkout bugfix
$git rebase upstream/master
As such, I would think that you shouldn't rebase any commits that were published by others. Rebasing makes good sense if you're working on a separate feature that you're not pushing yet, and you want to update your work in progress, however.
1
2
3
What would cause duplicate commits, if not rebasing? I should think that we want to avoid duplicate commits of this nature.
The reason I like to use rebase in these cases is because it is a first-come first-served kind of thing. If someone else gets their commits in first, and you are committing after them, then it's up to you to make sure your stuff works on top of their stuff. They were in first, their code is out there, you have to make yours work on top of theirs. Using rebase makes sure your commits come after all the other ones that came before you. Then everyone else who comes after you has to make sure their stuff works on top of yours.
Also, some of these crazy conflicts only occur when you start fetching stuff from multiple collaborators. I suggest that unless you really know what you are doing, that you only ever fetch from one upstream source, which would be me. If you always make sure your master branch is tracked to my master, and you rebase all of your stuff on top of it before making a pull request, then no weirdness like that can ever occur. The weirdness starts to happen when multiple people are sharing in different directions. For example.
my master - A B C
you master - A B C D
him master - A B C E
You and him share with each other and each do a little more work.
my master - A B C
you master - A B C D E F
him master - A B C E D G
Then you both send me pull requests. Ah crap!
If you don't know what you are doing, I highly suggest you only fetch from one place and rebase on top of that.
my master - A B C
you master - A B C D
him master - A B C E
I pull in changes as requested, and everyone else starts doing new work rebased on top of my master.
my master - A B C D E
you master - A B C D E F
him master - A B C D E G
EDIT: Ooh, I did it, but git rebase -i is scary.
Also, I'd like to point out another possible work pattern. Two people want to work on a particular feature together. What do they do? They should both create a separate branch, not just locally on their machines, but on GitHub. They can then work together on the feature in that separate branch. They can keep pulling each others changes just on that separate branch. Then when it's all good, one of them can merge it back into master, make a pull request, and it will disseminate.
Also, a lot of the 3D engines out there are really geared toward a more traditional idea of a 3D game. Their great for an FPS or a 3rd person shooter, but for a sim? I don't know.
We also have to consider the user interface. It's what DF is the worst at, so it's easy to beat it, but we should really hit it out of the park to deal a crushing blow. If I were playing DF I would think an isometric or actual 3D view, like the last two example screen shots, would make for a worse UI as you would have to rotate the camera often. I feel it should be more like Sim City, where you can rotate to four different angles, but you never really need to.